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ABSTRACT 


Proper measurement and management is necessary to effectively translate capability 
needs and technology opportunities into stable, affordable, and well-managed acquisition 
programs. The Technology Readiness Level (TRL) has proven to be the tool and process 
of choice for assessing the maturity of developing technologies within the Department of 
Defense (DoD). Yet, the TRL has proven incapable of consistently capturing the human- 
related aspects of technology development and their association with technology 
readiness. This thesis describes the initial development and evaluation of the Human 
Readiness Level (HRL). The purpose of the HRL framework is to complement TRLs in 
program risk management structures within the Integrated Defense Acquisition, 
Technology, and Logistics (IDAT&L) Life Cycle Management System. Further 
development and evaluation of the HRL framework is required beyond what was carried 
out as part of this thesis. However, the initial framework takes that first step towards 
providing acquisition professionals a comprehensive guide that ensures human-centric 
priorities are addressed throughout all phases and milestones of Defense Acquisition. 
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EXECUTIVE SUMMARY 


This thesis describes the development of an initial Human Readiness Level (HRL) 
framework, designed to complement the existing Technology Readiness Level (TRL) 
currently being used within the Integrated Defense Acquisition, Technology, and 
Logistics (IDAT&L) Life Cycle Management System. Technology maturity assessment 
tools, such as the TRL, serve as systematic measurement systems that are used as entry 
and exit criteria for transitioning milestones and are integral components to program risk 
management structures. Yet, the TRL has proven incapable of consistently capturing the 
human-related aspects of technology development and their association with technology 
readiness. 

The proposed HRL framework was developed with the aim of adding clarity to 
technical readiness assessments by emphasizing the socio-technical attributes of system 
development. Specifically, the HRL aims to reduce technology risks related to the human 
element by ensuring the adequate incorporation of Human Systems Integration (HSI) 
during technology maturity evaluations and by explicitly linking technology development 
to the effective design and specification of human-centered systems in Defense 
Acquisition. The HRL accomplishes this by providing a time and milestone-sensitive 
roadmap of activities that: a) detail critical organizational milestones (that ensure 
functional commitment to HSI); and b) define HSI domain-specific data collection and 
analysis, and a clear process for their management. The primary measure of HSI risk and 
maturity level is based on the execution and outcome of those milestone-sensitive 
management and analytical activities that have been designated in each progressive HRL. 

To evaluate the utility of the initial HRLs, two research efforts were carried out. 
In the first, an evaluation questionnaire was designed to gain feedback as to the overall 
worth and potential usefulness of the HRL framework. The questionnaire was given to 43 
subject matter experts (SMEs) in the fields of HSI and Defense Acquisition. A total of 15 
responses were obtained. Results of this evaluation indicated agreement among SMEs 
that the first iteration of the HRE concept, once fully developed, will serve as a valuable 
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HSI planning tool and complementary (HSI-specific) metric in program risk management 
structures. In order to improve the framework, many SMEs recommended expanding the 
HRL’s list of activities to account for a more diverse variety of programs that exist in 
acquisition. 

The second research effort was a case study regarding the usefulness of the HRL 
framework as applied to a specific acquisition program. Specifically, the HRL was 
evaluated in its ability to effectively contribute to the creation and sustainment of the 
acquisition program’s HSI Plan (HSIP) for the Ground Control Station Modernization 
(GMOD). Results of this SME evaluation suggested a general agreement that the HRL 
framework could serve as a beneficial tool in the development of an acquisition 
program’s comprehensive HSIP. Consistent with the findings from the first research 
effort, the SME did not feel the HRL framework contained all of the necessary HSI 
activities to account for all program needs. The feedback and lessons learned from both 
studies were documented and will serve to form a basis of advocacy for future HRL 
development efforts. 

Lurther development and evaluation of the HRL concept is required. However, 
the initial framework presented in this thesis takes that first step towards providing 
acquisition professionals a comprehensive guide that ensures human-centric priorities are 
addressed throughout all phases and milestones of Defense Acquisition. 


XVI 



ACKNOWLEDGMENTS 


I would like to express my sincere appreciation and gratitude to several 
individuals who have assisted me throughout the process of completing this thesis. I 
would first like to thank my thesis advisor, Lieutenant Commander Paul O’Connor, for 
providing me exceptional mentorship and guidance. Without his humor, good sense and 
judgment, this thesis effort would not have been possible. I want to thank my co-advisor. 
Dr. Hector Acosta, for having been unwavering in his support and advice throughout this 
entire endeavor. His passion and creativity would always light the road ahead. I would 
also like to thank my second reader. Dr. Michael McCauley, for his assistance in 
providing valuable insight and experience to my research. I extend my appreciation to 
various members of the 711th Human Performance Wing, Brooks City-Base, Texas. 
Their willingness to sponsor and coordinate efforts in data collection was essential in the 
timely completion of this thesis. 

Finally, I want to express my overwhelming love and appreciation for my family. 
My wife, Kristy, has been the most important source of support and encouragement. She 
is the love of my life and I honestly could not be the man I am today without her. To my 
daughters, Calla and Farrah, who are my ultimate sources of inspiration and happiness, I 
thank you. I love you both and will forever be proud of who you are. Lastly, and most 
especially, I thank the Lord Jesus Christ for blessing me with this life that I am so 
fortunate to live. 


xvii 



THIS PAGE INTENTIONALLY LEET BLANK 



I. 


INTRODUCTION 


A. PURPOSE 

This thesis develops and documents an executable process description of the 
Human Readiness Level (HRL) to complement the existing Technology Readiness Level 
(TRL) measurement process currently being used within the Integrated Defense 
Acquisition, Technology, and Logistics (IDAT&L) Life Cycle Management System. 
Technology maturity measurement tools, such as the TRL, serve as systematic 
measurement systems that describe the maturity of a particular technology and allow 
consistent comparison of maturity to be made between different types of technologies 
(Saucer, Verma, Ramirez-Marquez, & Gove, 2006). Throughout Department of Defense 
(DoD) acquisition programs, TRLs are used as entry and exit criteria for transitioning 
milestones and are integral components to program risk management structures. The 
proposed HRL measurement framework will add clarity to technical readiness 
assessments by emphasizing the socio-technical attributes of systems. The HRL aims to 
reduce technology risks related to the human element by ensuring the inclusion of Human 
Systems Integration (HSI) considerations during technology maturity evaluations. By 
capturing human-related components of evolving technologies, the acquisition and 
Science & Technology (S&T) communities will decrease operational and development 
risks linked to insufficient HSI. 

B. OVERVIEW 

The DoD relies heavily on the technological superiority of its weapon systems 
and armed forces to protect U.S. interests. Although the United States has produced some 
of the finest modem weapon systems in the world, the Government Accountability Office 
(GAO) points out that its acquisition programs often incur cost overruns, schedule delays, 
and performance shortfalls that weaken DoD’s buying power (GAO, 2006). This ongoing 
challenge is due in part to DoD’s difficulty in managing technology maturity. In 
numerous reports to Congressional Committees, the GAO has addressed the problems 
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with proceeding in system development with immature technologies and has detailed 
how best practices offer improvements to the way the DoD incorporates new technology 
into weapon system programs. According to a 2001 GAO report, “organizations that use 
best practices recognize that delaying the resolution of technology problems until 
product development—analogous to the Engineering and Manufacturing Development 
phase—can result in at least a ten-fold cost increase; delaying the resolution until after 
the start of production could increase costs by a hundred-fold” (GAO JSF, 2001, p. 1-2). 
The GAO went on to illustrate this point by providing the following comparison between 
the Joint Direct Attack Munition (JDAM) and the Comanche helicopter programs. 

For example, the Joint Direct Attack Munition used modified variants of 
proven components for guidance and global positioning. It also used 
mature, existing components from other proven manufacturing processes 
for its own system for controlling tail fin movements. The munition was 
touted for its performance in Kosovo and was purchased for less than half 
of its expected unit cost. However, the Comanche helicopter program 
began with critical technologies such as the engine, rotor, and integrated 
avionics at Technology Readiness Levels (TRLs) of 5 or below [see later 
for detailed discussion of TRLs]. That program has seen 101 percent cost 
growth and 120-percent schedule slippage as a result of these low 
maturity levels and other factors. (GAO JSF, 2001, p. 7) 

To enhance the ability of programs to select mature technologies for inclusion in 
their programs, the GAO recommended the use of TRLs (Graettinger et ah, 2002). The 
first published description of technology readiness can be found in a 1989 paper titled, 
“The NASA Technology Push towards Future Space Mission Systems.” In that paper, 
Stanley Sadin and his co-authors presented a seven-level readiness metric that assessed 
the risk associated with technology development. In the years that followed, the metric 
evolved into the nine levels that exist today. The TRL describes the maturity of 
technology elements with levels that begin in the earliest stages of scientific investigation 
(Level 1) to the successful use in the system (Level 9). Organizations, such as the Air 
Force Research Laboratory (AFRL) promotes TRLs as a means of evaluating the 
readiness of technologies to be incorporated into a weapon or other type of military 
system (Graettinger et ah, 2002). 
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Despite the practical utility of using TRLs to measure technology readiness, they 
fail to account for the critical human element. By inadequately considering the human 
capacity or requirements as part of the system, the ability to enhance human/system 
performance, maximize operational effectiveness, and reduce life cycle cost (LCC) is 
compromised (711 HPW/HPO, 2009). Therefore, additional methodologies are needed to 
ensure that human-centric priorities are implemented throughout all phases and 
milestones of defense acquisition. To this end, it is the goal of this thesis to provide the 
acquisition and S&T communities a human-centric approach to assessing technology 
maturity in order to avoid operational and development risks associated with insufficient 
consideration of the human element. 

C. NEED FOR HUMAN SYSTEMS INTEGRATION 

The level of technology that the U.S. provides to its armed forces is simply 
unparalleled. However, even with sophisticated technologies, optimized total systems 
performance still depends on the warfighter’s ability to use the technology fully and 
effectively (AFHSIO, 2009). HSI serves as the critical link that infuses human roles, 
structures, processes, and constraints into the design of systems to achieve total systems 
performance. By integrating HSI strategies throughout a system’s life cycle, programs are 
better able to identify, track and resolve human-related issues and ensure a balanced 
development of both human and technological capabilities (MoD HFI DTC, 2008). This 
is accomplished by incorporating functional areas, referred to as domains. These domains 
are identified in the DoD instruction on the operation of the Defense Acquisition System 
(DoDI 5000.02) and are illustrated in Table 1: 
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Table 1. 


HSI Domains and Definitions (After: 711 HPW/HPO, 2009) 


HSI Domain 

Definition 

Manpower 

The number and mix of personnel (military, civilian, and 
contractor) authorized and available to train, operate, 
maintain, and support each system acquisition. 

Personnel 

The human aptitudes, skills, knowledge, experience 
levels, and abilities required to operate, maintain, and 
support the system at the time it is fielded and 
throughout its life cycle. 

Training 

The instruction and resources required to provide 
personnel with requisite knowledge, skills, and abilities 
to properly operate, maintain, and support the system. 

Human Factors Engineering 

The comprehensive integration of human capabilities 
and limitations (cognitive, physical, sensory, and team 
dynamic) into system design, development, modification 
and evaluation to optimize human-machine performance 
for both operation and maintenance of a system. Human 
factors engineering designs systems that require minimal 
manpower, provide effective training, can be operated 
and maintained by users; and are suitable and 
survivable. 

Environment 

The factors concerning water, air, and land and the 
interrelationships which exist among and between water, 
air, and land and all living things. 

Safety 

The design and operational characteristics that minimize 
the possibilities for accidents or mishaps to operators 
which threaten the survival of the system. 

Occupational Health 

The design features that minimize risk of injury, acute 
and/or chronic illness, or disability, and/or reduced job 
performance of personnel who operate, maintain, or 
support the system. 

Survivability 

The characteristics of a system that reduce risk of 
fratricide, detection, and the probability of being 
attacked; and that enable the crew to withstand man¬ 
made or natural hostile environments without aborting 
the mission or suffering acute and/or chronic illness, 
disability, or death. 





















Factors of living and working conditions that is 
necessary to sustain the morale, safety, health, and 
comfort of the user population which contribute directly 
to personnel effectiveness and mission accomplishment, 
and often preclude recruitment and retention problems. 


Although HSI is a fundamental component of a total systems approach, the 
successful integration of HSI into systems engineering and acquisition life cycles 
continues to be a challenge. The Air Force 711th Human Performance Wing points to the 
following barriers that often exist in program development: 

• Lack of alignment of processes between the traditional practice of HSI and 
the human-based technical domains; 

• lack of alignment between human technical areas and the systems 
engineering and program management practices of DoD acquisition; and 

• inability to communicate, which causes an inability to share data or 
information throughout the acquisition phases and beyond (711 
HPW/HPO, 2008). 

Therefore, recognizing this need for HSI involvement and a sustainable human 
approach to system acquisition, there has been an international emergence of what is 
being termed, the Human View. The Human View was originally created to be used as a 
complementary element to the Operational, System and Technical Standards Views of the 
Ministry of Defence Architectural Framework (MODAF). The Human View aims to 
clarify the role of Human Factors Integration (HFI; the United Kingdom (UK) equivalent 
of HSI) when creating Enterprise Architectures in support of acquisition (MoD HFI DTC, 
2008). 

Architectures such as MODAF or the US-created DoD Architectural Framework 
(DoDAF) produce common systems engineering (SE) approaches to development, 
presentation, and integration of current and future systems. Within DoD, architectures 
are created for a number of reasons. Erom a practical perspective, experience has shown 

that the management of large organizations employing sophisticated systems in pursuit of 
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joint missions demands an organized, repeatable method for evaluating investments and 
investment alternatives, as well as the ability to effectively implement organizational 
change, create new systems, and deploy new technologies (DoD, 2007). Towards this 
end, the Human View connects HSI to DoDAF by supporting system design and 
development that effectively and affordably integrates human capabilities and limitations. 

Although, the Human View has been tailored to the major characteristics of 
architectural models, HSI practitioners working within Defense Acquisition have 
observed the potential for extended applications. An example of this extended 
application, which serves as the focus of this thesis, was conceived by Dr. Hector Acosta 
of the Air Force 711th Human Performance Wing (a co-advisor of this thesis). After 
recognizing the potential for using Human View technical details and attributes to 
structure HSI initiatives within technology readiness determinations. Dr. Acosta created 
the concept of the HRL. This thesis will build upon the work of Dr. Acosta by developing 
the HRL concept to form a defined and executable process description. This will allow 
HRLs to immediately become applicable to the acquisition and S&T communities. HRLs 
will facilitate the unification and integration of HSI processes within technology 
evaluations, to include formal and informal technology maturity assessments. Ultimately, 
HRLs will provide DoD acquisition programs the ability to reduce technology risks 
related to the human element and will help to maximize both return on investment (ROI) 
and system performance. 

D. SCOPE 

This thesis focused on defining Human Readiness Levels for use in technology 
development life cycles, using the detailed elements of the Human View. The tasks 
comprising this thesis included the following: 

• Conduct in-depth analysis of current DoD technology development 
processes and maturity standards (i.e.. Technology Readiness Assessment 
(TRA), TRL, etc.) adhered to in the Defense Acquisition System and 
document shortfalls in the consideration of human aspects of systems. 
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• Synthesize the technical details of the Human View and its relation to 
DoDAF and explicitly develop its linkage to the concept of an HRL. 

• Analyze the Human View/HRL amalgamation’s validity and usability in 
acquisition and S&T environments and begin the effort towards 
elaborating the HRL concept throughout the lifecycle of a system. 

E. THESIS ORGANIZATION 

A brief literature review of the significance and history of technology maturity is 
discussed in Chapter 11. This review encompasses the nine Technology Readiness Levels 
used by government and industry to measure technology maturity and program risk, and 
highlights the inability of the readiness levels to adequately capture human-specific 
elements of evolving technologies. To approach the issue, this thesis develops and 
defines the complimentary concept of the Human Readiness Level. Because the HRL 
employs the technical details of the Human View to structure HSI initiatives specifically 
within technology maturity assessments, the history and utilization of the Human View 
also is discussed in Chapter 11. The subsequent chapters are devoted to introducing and 
analyzing the HRL’s value, validity, and usability in acquisition and S&T environments. 
Chapter III details the development process behind the HRL and presents the first 
iteration of the HRL concept. Chapter IV discusses the initial evaluation of the HRL 
concept. This included Subject Matter Expert (SME) feedback that focused on accuracy, 
ease of use, and completeness of the HRE framework. Chapter V describes the 
succeeding validation of the HRE model being practically applied in a brief case study. 
Chapter VI summarizes this thesis effort and discusses future implications for HRE use in 
Defense Acquisition. 
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II. LITERATURE REVIEW 


A. TECHNOLOGY 

1. Technology Defined 

The National Aeronautics and Space Administration (NASA) Technology Plan 
defines technology as the practical application of knowledge to create the capability to do 
something entirely new or in an entirely new way. This can be contrasted to scientific 
research, which encompasses the discovery of new knowledge from which new 
technology is derived, and engineering that uses technology derived from this knowledge 
to solve specific technical problems (Bilbro, 2007). Similarly, Dictionary.Com defines 
technology as follows: 

The branch of knowledge that deals with the creation and use of technical 
means and their interrelation with life, society, and the environment, 
drawing upon such subjects as industrial arts, engineering, applied 
science, and pure science; a technological process, invention, method, or 
the like; the sum of the ways in which social groups provide themselves 
with the material objects of their civilization. (Technology, n.d.) 

Science and technology are terms that are often seen as one and the same. 
However, William Nolle (2008) points out in his book, ‘Did 1 Ever Tell You about the 
Whale? Or Measuring Technology Maturity’, that science and technology are related, but 
not identical. He states that science is the search for new knowledge for knowledge’s 
sake, while technology takes that knowledge of science and makes it do something useful 
for society. Therefore, when scientific knowledge is put to use, we have technology. The 
form in which technology takes shape can be seen in hardware (i.e., cell phone, 
computer, aircraft, etc.) or in non-material technologies (i.e., software, new practices or 
procedures; Nolle, 2008). This understanding of what technology is and what it may 
encompass will serve as the contextual foundation for the remaining discussions in this 
chapter’s literature review. 
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The concept of technology maturity and how it is currently measured in 
government and industry will first be examined below, followed by the limitations 
inherent in such measurements. After which, the discussion focuses on the need for a 
human approach to technology readiness and describes the resulting technological 
immaturity that occurs when the human is not adequately incorporated. This chapter ends 
with the emergence of the Human View and details the ability of the Human View to 
integrate and shape human-specific elements into the design and implementation of 
technology. 

2. Technology Maturity 

With an understanding of what technology is and what it may encompass, it is 
appropriate to now examine what technology maturity might mean and how it is currently 
measured. In Dr. Tom Cruse’s 2006 report, "Multi-Dimensional Assessment of 
Technology Maturity' he points out that the term, maturity implies aging, or growth. With 
technology growth, Cruse states that the technology itself does not necessarily change, 
but rather our understanding of the technology changes. He explains that as our 
understanding improves or grows, the technology’s usefulness should improve as well. 
Therefore, the more we learn about a technology, the better we can apply it to meet our 
needs. For the most part, the customer or end user of a technology intuitively knows what 
it means for a technology to mature because they will often wait for a new, immature 
product to “get the bugs out” before purchasing. The expectation for a technology to 
become more mature, more capable of meeting needs and requirements after it “grows 
up” is the basic premise of technological maturity (Nolte, 2008). 

Within the DoD, the evolution and management of technology maturity continues 
to be a subject of study and emphasis. In their work on best business practices in the last 
two decades, the GAO studied a number of leading commercial firms to determine key 
factors in successful product development. They reported that one such key factor is 
maturing a new technology far enough to get it into the right size, weight, and 
configuration needed for the intended product. After this is demonstrated, the technology 
is said to be at an acceptable level for product development (Graettinger et ah, 2002). 
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According to the GAO, no element is more important than having technology 
advanced enough to meet requirements, but also mature enough to be predictably 
managed and available at the start of the product development cycle. ” They further 
stated that, “maturing new technology before it is included on a product is perhaps the 
most important determinant of the success of the eventual product—or weapon system ” 
(GAO, 1999, p. 12). 

To subjectively quantify the maturity of certain technologies and to ensure that 
new technologies are vigorously pursued and successfully moved into program 
development, the adoption of a disciplined and knowledge-based method is often 
recommended (GAO, 1999). The beginnings of such a method can be traced back as 
early as 1969, where NASA articulated the status of new technology planned for use in 
future space systems. The examples used were between the then-established practice of 
the “flight readiness review”, and a new idea through which the level of maturity of new 
technologies could be assessed - the “technology readiness review.” 

Although NASA used the emerging concept of technology readiness for some 
time after, the actual readiness level scale was not devised and published until 1989. That 
year, in an effort to develop a “systems-technology model,” Mr. Stan Sadin of the Office 
of Aeronautics and Space Technology codified seven levels of technology readiness. The 
seven readiness levels showed how the more traditional breakdown of research and 
development were incorporated into four loosely defined categories (Nolte, 2008): basic 
research; feasibility; development; and demonstration 

These readiness levels marked a significant change in emphasis on the part of 

NASA where technology had previously been viewed as merely having a supporting role. 

This change in role for technology was the result of a revision in the National Space 

Policy stating that NASA’s technology program “... shares the mantle of responsibility 

for shaping the Agency's future... ” The new emphasis on technology’s responsibility was 

cause for a reassessment of how technologies were developed and infused, with a goal of 

approaching technology development and infusion in a much more systematic way to 

increase the likelihood of success (JB International, n.d.). As Sadin, Povinelli, and Rosen 

(1989) point out, when historical records are analyzed, it can be demonstrated that the 
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difference between success and failure is traceable to the adequacy, or depth, of the 
advanced research and technology program pursuing technology readiness. 

Within DoD, the concept of technology readiness was not entirely embraced until 
1999. That year, the GAO conducted an assessment of how best practices offer 
improvements to the way DoD incorporates new technologies into Defense Acquisition 
programs. The GAO concluded that “the experiences of DoD, and the commercial 
technology development cases GAO reviewed, indicate that demonstrating a high level of 
maturity before new technologies are incorporated into product development programs 
puts those programs in a better position to succeed” (GAO, 1999, p. 3). Soon after, in a 
2001 memorandum, the Deputy Undersecretary of Defense for Science and Technology 
officially endorsed the use of the Technology Readiness Level (TRL) as the standard 
format for measuring technology maturity in new major programs within Defense 
Acquisition (Graettinger et ah, 2002). 

TRLs follow a scale from 1 {basic research) to 9 (system operation). A 
technology determined to be at TRL 1 is at the lowest level of technology readiness, 
where “scientific research begins to be translated into applied research and development” 
(GAO, 2006, p. 56). By the time the technology has reached a TRL 9, the technology has 
progressed through formulation of an initial concept for application, proof of concept, 
demonstration in a laboratory environment and realistic environment, and integration into 
a system, and has been “system qualified” and then “system proven.” This last state of 
development, where the technology is operating under actual mission conditions, is TRL 
9 (Valerdi, 2004). Table 2 provides the DoD’s original technology readiness levels and 
descriptions from a systems perspective with supplemental definitions provided in Table 
3 (separate matrices for DoD’s hardware and software TRLs are detailed extensively in 
Appendices A and B). 
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Table 2. 


DoD Technology Readiness Levels (After: GAO, 2006) 


Technology Readiness Level 

Description 

1. Basic principles observed and reported. 

Lowest level of technology readiness. 
Scientific research begins to be translated 
into applied research and development. 
Examples might include paper studies of a 
technology’s basic properties. 

2. Technology concept and/or application 
formulated. 

Invention begins. Once basic principles are 
observed, practical applications can be 
invented. Applications are speculative and 
there may be no proof or detailed analysis 
to support the assumption. Examples are 
still limited to analytic studies. 

3. Analytical and experimental critical 
function and/or characteristic proof of 
concept. 

Active research and development is 
initiated. This includes analytical studies 
and laboratory studies to physically 
validate analytical predictions of separate 
elements of the technology. Examples 
include components that are not yet 
integrated or representative. 

4. Component and/or breadboard. 
Validation in laboratory environment. 

Basic technological components are 
integrated to establish that they will work 
together. This is relatively “low fidelity” 
compared to the eventual system. 
Examples include integration of “ad hoc” 
hardware in a laboratory. 

5. Component and/or breadboard validation 
in a relevant environment. 

Eidelity of breadboard technology 
increases significantly. The basic 

technological components are integrated 
with reasonably realistic supporting 
elements so it can be tested in a simulated 
environment. Examples include “high 
fidelity” laboratory integration of 

components. 

6. System/subsystem model or prototype 
demonstration in a relevant environment. 

Representative model or prototype system, 
which is well beyond that of TRE 5, is 
tested in a relevant environment. 
Represents a major step up in a 
technology’s demonstrated readiness. 
Examples include testing a prototype in a 
high-fidelity laboratory environment or in 
simulated operational environment. 
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7. System prototype demonstration in an 
operational environment. 

Prototype near, or at, planned operational 
system. Represents a major step up from 
TRL 6, requiring demonstration of an 
actual system prototype in an operational 
environment such as in an aircraft, vehicle, 
or space. Examples include testing the 
prototype in a test bed aircraft. 

8. Actual system completed and qualified 
through test and demonstration. 

Technology has been proven to work in its 
final form and under expected conditions. 
In almost all cases, this TRL represents the 
end of true system development. Examples 
include developmental test and evaluation 
of the system in its intended weapon 
system to determine if it meets design 
specifications. 

9. Actual system proven through successful 
mission operations. 

Actual application of the technology in its 
final form and under mission conditions, 
such as those encountered in operational 
test and evaluation. Examples include 
using the system under operational mission 
conditions. 


Table 3. Additional Definitions of TRL Descriptive Terms (From: DoD, 2009) 


Term 

Definition 

Breadboard 

Integrated components that provide a 
representation of a system/subsystem and 
that can be used to determine concept 
feasibility and to develop technical data. 
Typically configured for laboratory use to 
demonstrate the technical principles of 
immediate interest. May resemble final 
system/subsystem in function only. 

High Eidelity 

Addresses form, fit, and function. A high- 
fidelity laboratory environment would 
involve testing with equipment that can 
simulate and validate all system 
specifications within a laboratory setting. 

Low Eidelity 

A representative of the component or 
system that has limited ability to provide 
anything but first-order information about 
the end product. Low-fidelity assessments 
are used to provide trend analysis. 
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Model 

A functional form of a system, generally 
reduced in scale, near or at operational 
specification. Models will be sufficiently 
hardened to allow demonstration of the 
technical and operational capabilities 
required of the final system. 

Operational Environment 

Environment that addresses all the 
operational requirements and specifications 
required of the final system to include 
platform/packaging. 

Prototype 

A physical or virtual model used to 
evaluate the technical or manufacturing 
feasibility or military utility of a particular 
technology or process, concept, end item, 
or system. 

Relevant Environment 

Testing environment that simulates both 
the most important and most stressing 
aspects of the operational environment. 

Simulated Operational Environment 

Either (1) a real environment that can 
simulate all the operational requirements 
and specifications required of the final 
system or (2) a simulated environment that 
allows for testing of a virtual prototype. 
Used in either case to determine whether a 
developmental system meets the 

operational requirements and specifications 
of the final system. 


To illustrate how technologies progress from initial concept to proven 
performance within the readiness levels, the following hypothetical example about an 
airborne communications radio is provided: 

First, the idea for a new radio is conceived. The idea reaches TRL 3 when 
analytical studies and some tests of the technology’s elements, such as a 
circuit, back it up. When initial hand-built versions of all of the radio’s 
basic elements are connected and tested together, the radio reaches TRL 
5. This is sometimes referred to as a “breadboard” article; although it 
may function like a radio, it does not look like one because the individual 
parts are attached to plywood and hand-wired together. When the 
technology is built into a generic model, which is well beyond the 
breadboard tested in TRL 5, and demonstrated in a laboratory 
environment, the radio reaches TRL 6. This model represents the last level 
of demonstration before the radio becomes tailored for application to a 
specific aircraft. When the components are assembled inside a case that 
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resembles the final radio design and are demonstrated aboard a surrogate 
for the intended aircraft, the radio reaches TRL 7. TRL 8 is reached when 
the radio is put in its final form, installed in the intended aircraft’s 
cockpit, and tested in conjunction with the other aircraft equipment with 
which it must interface. TRL 9 is achieved when the radio is successfully 
operated on the aircraft through several test missions. (GAO, 1999, p. 22- 
23) 

The TRL offers many benefits to the S&T and acquisition communities. For 
example, they provide a snapshot of a technology’s maturity at a given instant and can be 
used to establish benchmarks for maturity goals. The TRL can also serve as a 
communication device in those cases where a technology developer must hand off a 
technology to another organization for product or system development. TRLs can assist 
both sides in understanding exactly what each is required to do. By providing a common 
reference point for the technology developer and user, TRLs assist in eliminating 
misunderstandings and ambiguities in the technology transition process (Nolte, 2008). 

B. LIMITATIONS OF THE TRL 

Despite the practical utility of using TRLs to measure technology readiness, they 
do, however, have significant limitations. For instance, author Jim Smith (2005) states 
that TRL definitions often combine several different aspects of technology readiness into 
one metric, thus “blurring” the different contributors to readiness. An example of this can 
be seen in how the U.S. Army Communications Electronics Command (CECOM) defines 
a TRE 7 in their draft software TRE scale (Graettinger et ah, 2002): 

Represents a major step up from TRL 6, requiring demonstration of an 
actual system prototype in an operational environment... Algorithms run 
on the processor of the operational system and are integrated with actual 
external entities. Software support structure is in place. Software releases 
are in distinct versions. Frequency and severity of software deficiency 
reports do not significantly degrade functionality or performance. 

This description combines the functionality of algorithms, the maintainability of a 
software support structure, and the reliability of software deficiency reports to indicate a 
specific level of readiness at TRE 7. The problem, however is that the manner in which 
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these factors are combined makes it difficult to understand how any one aspect influences 
the overall readiness of the technology (Smith, 2005). 

Similarly, William Nolle states: 

The TRL scale measures maturity along a single axis, the axis of 
technology capability demonstration. A full measure of technology 
maturity, or in the commercial world, product maturity, would be a multi¬ 
dimensional metric. It’s not uncommon to find references to 12 or more 
dimensions of product or technology maturity. One writer speaks of 16 
different dimensions of maturity. The TRL measures only one of the 16. 
(Hobson, 2006, p. 2) 

In an attempt to provide a more complete measure of technology maturity within 
the Defense Acquisition Management System, the GAO (2006) recommended that DoD 
expand its use of Technology Transition Agreements (TTAs) and metrics to cover aspects 
of technology maturity not explicitly attended to by the TRL (Appendix C summarizes 
the management of technology maturity in the Defense Acquisition Management 
System). Within this recommendation, the GAO specifically emphasized the utilization 
of other maturity metrics, such as the Manufacturing Readiness Level (MRL) to support 
better management of technology risk and to avoid the pitfalls of proceeding with 
immature technologies (GAO, 2006; Appendix D details the impact of immature 
technologies). Although they were only recently introduced, MRLs have already gained 
wide acceptance throughout government and industry (AFRL/RXM, 2009). 

In recent years, many readiness levels like the MRL have surfaced. Other 
examples of readiness levels developed in varying degrees include the Design Readiness 
Level, Integration Readiness Level, and System Readiness Level. Ultimately, the efforts 
towards creating alternative readiness levels have been to replace, amplify, or 
compliment existing TRL definitions to better measure and manage specific 
programmatic areas of concern (Sauser, Ramirez-Marquez, Magnaye, & Tan, 2008). 

C. A HUMAN-CENTERED APPROACH IS NEEDED 

While there continues to be an increasing number of TRL variants being created, 
there has not, however, been any specific effort towards capturing the socio-technical 
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attributes of technology and its association with technology readiness. Without a human- 
centered focus, the Human Factors Integration (HFI) Defence Technology Centre (DTC) 
within the Ministry of Defence (MoD) states that programs are ill-prepared to identify, 
track, and resolve human-related issues that would ensure the balanced development of 
both human and technological capabilities (MoD HFI DTC, 2008). 

Historically, the lack of appropriate Human Systems Integration (HSI; the U.S., 
Canadian, and Australian equivalent to HFI) has been a significant stumbling block for 
many programs, even those with supposedly mature technologies. Whether it is 
redesigns, substandard system performance, or dangerous system failures endangering 
life and equipment, poor HSI involvement in technology maturation can be catastrophic 
(Technology Development (TD) 1-12 Implementation Team, 2008). Two unfortunate, yet 
classic examples of this can be seen in Table 4: 

Table 4. Three Mile Island Incident / Loss of the Mars Climate Orbiter (After: 

Technology Development (TD) 1-12 Implementation Team, 2008) 

Three Mile Island Incident _ 

On March 28, 1979, operators at Three Mile Island, a nuclear power plant in 

Pennsylvania, made a series of mistakes that led to a near meltdown of the plant’s reactor 

core. This accident was caused by a cascade of equipment failures and operator errors. 

The result of the accident was a release of approximately 1200 millirem/hour of radiation 

into the environment, forcing the evacuation of several thousand residents of the 

surrounding area. Fortunately, there were no deaths as a direct result of the incident. The 

near meltdown of the reactor occurred when an emergency relief valve at the top of the 

pressurizer failed to close, resulting in a loss of the pressurizer steam bubble and a loss of 

reactor control system pressure and quantity. The indicator lights on which the operators 

were relying to determine the position of the relief valve led them to believe that the 

valve was closed. However, the indicator light was not displaying the actual system 

state—rather; it was showing the presence of a signal commanding the valve to close. In 

other words, the operators believed the relief valve was closed when in reality the valve 

was open, though it had been commanded to close. This led the system operators to 

conclude falsely that a leak had occurred, and they began to act accordingly. 
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The operators continued to make errors that increased the volatility of the system, 
such as confusing reactor B with reactor A (a problem directly attributable to the control 
panel layout). Not until two hours into the accident, when an operator who had recently 
arrived finally realized that the relief valve was at fault and performed the proper actions 
to correct the problem. In the end, the investigation by the Nuclear Regulatory 
Commission into the human factors (HF) aspects of the accident determined that the 
human errors which occurred during the incident were not due to operator deficiencies 
but rather to inadequacies in equipment design, information presentation, emergency 
procedures, and training. 

Loss o f the Mars Climate Orbiter (MCO) _ 

The Mars Climate Orbiter (MCO) Mission objective was to orbit Mars as the first 

ever interplanetary weather satellite and provide a communications relay for the Mars 
Polar Lander (MPL). The MCO was launched on December 11, 1998, and was lost 
sometime following the spacecraft's Mars Orbit Insertion (MOI) maneuver. The 
spacecraft's carrier signal was last seen at 09:04:52 UTC on Thursday, September 23, 
1999. 

During the 9 month journey from Earth to Mars, propulsion maneuvers were 
periodically performed to remove angular momentum buildup in the on-board reaction 
wheels (flywheels). These Angular Momentum Desaturation (AMD) events occurred 10- 
14 times more often than was expected by the operations navigation team. This was 
because the MCO solar array was asymmetrical relative to the spacecraft body as 
compared to Mars Global Surveyor (MGS) which had symmetrical solar arrays. This 
asymmetric effect significantly increased the Sun-induced (solar pressure-induced) 
momentum build-up on the spacecraft. Additionally, the angular momentum (impulse) 
data was in English, rather than metric, units. This measurement confusion resulted in 
small errors being introduced in the trajectory estimate over the course of the 9 month 
journey. At the time of Mars insertion, the spacecraft trajectory was approximately 170 
kilometers lower than planned. As a result, MCO either was destroyed in the atmosphere 
or re-entered heliocentric space after leaving Mars' atmosphere. The Board recognized 
that mistakes occur on spacecraft projects. However, sufficient processes are usually in 
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place to catch these errors before they become critical to mission success. Unfortunately 
for MCO, the root cause was not caught by the processes in-place in the MCO project. 

The MCO Mishap Investigation board (MIB) determined that the root cause of the 
loss of the MCO spacecraft was the failure to use metric units in the coding of a ground 
software file, "Small Forces", used in trajectory models. Specifically, thruster 
performance data in English units instead of metric units was used in the software 
application code titled SM_FORCES (small forces). The AMD file contained the output 
data from the SM_FORCES software. The data in the AMD file was required to be in 
metric units per existing software interface documentation, and the trajectory modelers 
assumed the data was provided in metric units per the requirements. Inadequate employee 
and programmer training were cited as the reasons for the omission of the English-to- 
metric conversion factor in the software program used to generate the AMD files. 


The two examples described above highlight how human errors can result in 
disastrous events, particularly when they are synergistically compounded. Therefore, to 
reduce errors, maximize performance, and enhance safety, proper human-centered design 
must be accomplished. This effort must attend to all areas that impact the human in the 
system, including training, source selection, human-computer interaction, ergonomics, 
safety, survivability, habitability, quality of life, and human performance in extreme 
environments (e.g., space, underwater, and Polar expeditions; O’Connor & Cohn, 2010). 

The Air Force 711th Human Performance Wing states clearly that if humans, as 
part of an integrated human-technology system, are not considered in the design and 
implementation, the system may not achieve the desired operational capability (711 
HPW/HPO, 2009). Expanding on this belief, the UK Vice Admiral Sir Jeremy Blackham 
stated: 


Capability is not just a function of equipment performance, but depends on 
a combination of interacting elements. Some of the most difficult issues to 
address lie in the human factors area. The types of systems we are 
specifying and procuring now will shape the roles, responsibilities and 
career paths of future servicemen and women. They will also have to be 
operated in very demanding circumstances of fatigue, hunger, stress and 
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even fear, by the sort of men and women we recruit. They will therefore 
determine not just the working environment of our people, but ultimately 
their utility in these harsh conditions will determine our operational 
success and our ability to retain the right people. (MoD, 2007, p. 4) 

Marine Corps General James Mattis, head of Joint Forces Command made a 
similar point while addressing a joint war-fighting conference. Calling on industry to 
focus on human-centered design, General Mattis was quoted saying, .. if what you ’re 
doing is going to enable the human interface, then you ’re on the right track... if not, you 
don’t want things that take geniuses on the battlefield to operate. We need to create 
systems and organizations and equipment that don’t need a master’s degree in math” 
(Cavas, 2010, para. 8). 

With today’s increasingly interconnected, diverse, and distributed work 
environments, the focus of human-centered design and technology development is more 
important now than ever. Current DoD acquisition policy requires optimizing total 
system performance and minimizing the cost of ownership through a “total system 
approach” to acquisition management (DoDD 5000.01, 2003). As seen in the Three Mile 
Island incident, as well as the loss of the Mars Climate Orbiter, the total system includes 
not only the prime mission equipment, but also the people who operate, maintain, and 
support the system; the training and training devices; and the operational and support 
infrastructure (DAU, 2009). The DoDI 5000.02, which details the operation of the 
Defense Acquisition System states: 

The program manager (PM) shall have a plan for HSI in place early in the 
acquisition process to optimize total system performance, minimize total 
ownership costs, and ensure that the system is built to accommodate the 
characteristics of the user population that will operate, maintain, and 
support the system. (DoDI 5000.02, 2008, p. 60) 

In addition. Chapter 6 of the Defense Acquisition Guidebook (DAG) states: 

The HSI domains (manpower, personnel, training, environment, safety and 
occupational health, human factors engineering, survivability, and 
habitability) can and should be used to help determine and work the 
science and technology gaps to address all aspects of the system 
(hardware, software, and human). The program manager should integrate 
system requirements for the HSI domains with each other, and also with 
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the total system. As work is done to satisfy these requirements, It is vitally 
important that each HSI domain anticipate and respond to changes made 
by other domains or which may be made within other processes or 
imposed by other program constraints. (DAU, 2009, section 6.2, para. 1). 

Even with such guidance, incorporating HSI into technology and system life 
cycles early continues to be a challenge. The policies that are currently set forth 
unfortunately address HSI as an act rather than a process. DoDI 5000.02, for example, 
thoroughly defines and details the five phases of the Defense Acquisition Life Cycle (see 
Figure 4 in Appendix C); however it does not fully integrate or define HSI’s role within 
each of those phases. In addition, there is no solid guidance on the preferred sequencing 
of needed HSI activities, nor are there any HSTspecific task requirements to support 
adequate accountability and implementation. Without an appropriate HSI process in 
place, the consistent management and mitigation of human-related risk inherent with all 
developing technologies and systems will continue to be a challenge. 

D. EMERGENCE OF THE HUMAN VIEW 

Recognizing the abovementioned need for HSI involvement and a sustainable 
human approach to system acquisition, there has been an international emergence of what 
is being termed, the Human View. The Human View is a supplementary view to existing 
Architectural Frameworks, such as the DoD Architectural Framework (DoDAF) that 
explicitly models the human elements being shaped in the process of capability design. 
By doing so, HSI is considered early and related closely to the design and 
implementation of technology (MoD HFI DTC, 2008). The ultimate purpose of the 
Human View is to enable effective HSI processes within the design of complex, large- 
scale, socio-technical systems. 

Architecture, as defined by DoD, is the structure of components, their 

relationships, and the principles and guidelines governing their design and evolution over 

time (DoDAF VI.5, 2007). Architecture frameworks, such as DoDAF are used within 

engineering and acquisition communities to produce common approaches to 

development, presentation, and integration of current and future systems. The DoDAF 

provides the guidance and rules for developing, representing, and understanding 
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architectures based on a common denominator across DoD, Joint, and multinational 
boundaries. It provides insight for external stakeholders into how the DoD develops 
architectures. The DoDAF is intended to ensure that architecture descriptions can be 
compared and related across programs and mission areas, while establishing the 
foundation for analyses that supports decision-making processes throughout the DoD 
(DoDAF VI.5, 2007). 

Within DoD, architectures are created for a number of reasons. From a practical 
perspective, the management of large organizations employing sophisticated systems, 
technologies, and services in pursuit of complex, joint missions demands a structured, 
repeatable method for evaluating investments and investment alternatives. Architectures 
help to implement organizational change effectively, create new systems, deploy new 
technologies, and offer services which add value to decisions and management practices 
(DoDAF V2.0, 2009). Newer architecture framework versions, such as DoDAF V2.0 
address Net-centric, System of Systems, and System/Services concepts while 
emphasizing data-centric processes to support DoD managers (as process owners and/or 
decision-makers). 
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Figure 1. Architecture Viewpoints in DoDAF V2.0 (From: DoDAF V2.0 Vol. 1, 2009) 
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Figure 1 provides a graphical representation of the different perspectives or 
viewpoints that logically combine to describe system architectures in DoDAF V2.0. It is 
important to note that a view is only a presentation of a portion of the total architectural 
data, in the sense that a photograph provides only one view of the object within the 
picture, not the entire representation of that object. The viewpoints are organized into 
eight basic sets with each set depicting certain architecture attributes as described in 
Table 5: 

Tables. DoDAF V2.0 Viewpoint Attributes (After: DoDAF V2.0 Vol. 1,2009) 


Viewpoint 

Architecture Attributes 

All Viewpoint 

Provides overarching descriptions of the 
entire architecture and defines the scope and 
context of the architecture 

Capability Viewpoint 

Captures the goals associated with the overall 
vision for executing a specified course of 
action, or the ability to achieve a desired 
effect under specific standards and conditions 
through combinations of means and ways to 
perform a set of tasks 

Data and Information Viewpoint 

Captures business information requirements 
and structural business process rules, and it 
describes the information that is associated 
with information exchanges 

Operational Viewpoint 

Captures the organizations, tasks, or 
activities performed, and information that 
must be exchanged between them to 
accomplish DoD missions 

Project Viewpoint 

Captures how programs are grouped in 
organizational terms as a coherent portfolio 
of acquisition programs 

Services Viewpoint 

Captures system, service, and 

interconnection functionality providing for, 
or supporting, operational activities 

Standards Viewpoint 

The minimal set of rules governing the 
arrangement, interaction, and 

interdependence of system parts or elements 

Systems Viewpoint 

Captures the information on supporting 
automated systems, interconnectivity, and 
other systems functionality in support of 
operating activities 
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The above viewpoints do not, however, adequately portray the human as a unique 
part of the system, nor do they capture the human-centered design aspects needed to 
ensure the effectiveness of human operated systems, such as users’ requirements, or 
capabilities and limitations (Handley & Smillie, 2009). It is due to this deficiency that the 
Human View was created. 

The North Atlantic Treaty Organization (NATO) Human View Handbook states 
that the Human View forms a basis for decisions by stakeholders by providing a 
structured linkage from the engineering community to the manpower, personnel, training, 
and human factors engineering communities. It provides a way to integrate HSI into the 
mainstream acquisition and systems engineering process by promoting early and often 
consideration of human roles. Additionally, it provides early coordination of task analysis 
efforts by both system engineering and HSI teams. Table 6 provides an overview of 
NATO’s current set of Human View products (greater detail of all Human View products 
and their potential elements can be found in Appendix E). 

Table 6. NATO Human View Products Overview (After: Handley & Smillie, 

2009) 


Human View 

Description 

HV-A: Concept 

A conceptual, high-level representation of the 
human component of the enterprise architecture 
framework. 

HV-B: Constraints 

Sets of characteristics that are used to adjust 
the expected roles and tasks based on the 
capabilities and limitations of tbe human in the 
system. 

HV-C: Tasks 

Descriptions of the human-specific activities in 
the system. 

HV-D: Roles 

Descriptions of the roles that have been defined 
for the humans interacting with the system. 

HV-E: Human Network 

The human to human communication patterns 
that occur as a result of ad hoc or deliberate 
team formation, especially teams distributed 
across space and time. 

HV-F: Training 

A detailed accounting of how training 
requirements, strategy, and implementation 
will impact the human. 
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Figure 2. NATO Human View Product Relationships (After: Handley & Smillie, 2009) 


The NATO Human View and its set of products facilitate design decisions by 
identifying relevant elements or specific sets of data. Relationships can be defined 
between the products that express dependencies between the data. Figure 2 depicts some 
of the relationships that have been established between the Human View products 
(Handley & Smillie, 2009). 

The approach NATO has taken with their Human View framework has 
emphasized the explicit need of merging seamlessly and efficiently with good systems 
engineering practice. It establishes a logical and systematic framework for specifying the 
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professional data collection and analysis actions needed for HSI implementation and it 
makes explicit human, crew, and team socio-behavioral processes as integral to total 
system performance. 

An example of the Human View products being used to capture human elements 
can be seen in a case study based on a subset of the U.S. Army’s Future Combat System 
(FCS). The FCS program was a modernization initiative designed to link soldiers to a 
wide range of weapons, sensors, and information systems by means of a mobile ad hoc 
network architecture that enables joint interoperability, shared situational awareness, and 
the ability to execute highly synchronized mission operations. It consists of the network, 
the soldier, and 14 systems, including eight manned ground vehicles, two classes of 
unmanned aerial systems, two classes of unmanned ground systems, unattended ground 
sensors, and the Non-Line of Sight Launch System (Handley & Smillie, 2009). Table 7 
indicates the type and scope of information captured in each of the Human View products 
for the FCS example. 
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Table 7. Content of Human View Products for PCS Example (From: Handley & 

Smillie, 2009) 


Human View Products 

KCS Case Study Products Description 

HV-A Concept 

A graphic that shows a soldier connected through a network to 14 
systems. This repre.scnts the FCS "One Soldier. One Network, 14 
Systems” concept. 

HV-B Personnel Constraints: 
Manpowcr Projections (HV-B 1) 

A matri.\ that shows the distribution of the 50 personnel in an Infantry 
Platoon by both the 12 identilled roles and across the five platoon 
vehicles. 

HV-B Human Factor 

Constraints: Health Hazards 
(HV-B5) 

A table that show s the operating limits of infantry personnel to noise 
and heat, as well as rest requirements. 

HV-C Tasks 

A table that decomposes the platoon level tasks of Tactical Road 
March and Reaction to Ambush to 15 squad level tasks. 

HV-D Roles 

A table listing the 12 roles defined for the Infantry Platoon and their 
required Military Occupational Specialty (MOS). An additional table 
indicates the assignment of the roles to the previously defined tasks. A 
third table indicates the controllers and system interfaces available to 
each role. 

HV-F. Human Network 

A matrix that indicates the different team compositions within the 
platoon, both by vehicle and by functionality; the type of interactions 
i.e., collaboration, coordination, and supers ision, are also indicated. 

For example, w ithin three of the vehicles are two rifle squads lead by a 
team leader, with supervision from a squad leader. Across the vehicles 
are the drivers w ho coordinate maneuvers. 

HV-F Training 

A tabic that indicates the current training level for each MOS 
type indicated in the role table, and the additional training 
required to attain the knowledge, skill, and abilities to operate 
the FCS systems. 

HV-G Metrics 

A listing of the platoon level performance metrics for conducting 
Counter .Ambush Operations, as well as individual squad level 
performance standards based on FCS specifications. 


The subset of the ECS that was analyzed included the capabilities and limitations 
of an infantry platoon and the resources available in the Infantry Carrier Vehicle (one of 
the 14 ECS systems). The case study focused on the tasks involved during a tactical road 
march and the platoon’s reaction to an enemy ambush. As illustrated above, the Human 
View products were able to capture the different roles the platoon members assumed 
during these operations (i.e., platoon leader, driver, etc.), the interactions between the 
platoon functional teams, and the ECS technology interfaces. Ultimately, a fully 
populated and developed Human View would reflect the product of iterative analyses 


28 
















describing details of human sensory, perceptual, emotional, social, cognitive, ergonomic, 
biomechanical, motor, communicative, and decisional processes consistent with man- 
equipment interactions. 

E. SUMMARY 

In order to effectively translate capability needs and technology opportunities into 
stable, affordable, and well-managed acquisition programs, proper measurement and 
management of technology maturity must occur. The TRL has proven to be the tool of 
choice for describing the maturity of developing technologies. Yet, due to fundamental 
limitations, the TRL has been incapable of capturing the human-related attributes of 
technology and its association with technology readiness. 

The need for a better fusion of human-centric elements is not, however, unique to 
only technology maturity management. This issue continues to surface in all phases of the 
acquisition life cycle and it is in that very reason the international community created the 
Human View. The Human View, which is a complimentary view to existing architectural 
frameworks, establishes a logical and systematic structure for specifying the professional 
data collection and analysis actions needed for HSI implementation. 

The next chapter introduces the first iteration of the proposed Human Readiness 
Level and its attempt towards applying the technical details of the Human View and other 
existing sources to structure HSI initiatives specifically within technology maturity 
assessments. The steps taken in developing the HRL concept will first be discussed, 
followed by the initial validation studies in subsequent chapters. 
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III. HUMAN READINESS LEVEL DEVELOPMENT 


A. OVERVIEW 

In Defense Acquisition risk management structures, developing technologies are 
monitored and managed for associated risk by way of Technology Readiness 
Assessments (TRAs) and their assignment of TRLs (see Appendix C for more details 
regarding the TRA process). The lower the level of technology readiness, the more 
ground must be covered to bring the technology to the point at which it can meet the 
intended product’s cost, schedule, and performance requirements with little risk. As 
mentioned in Chapter II, the TRL does not adequately capture socio-technical elements 
of developing technology and without such attention; risk related to the human will 
continue to exist. This chapter describes the development of the first iteration of the HRL. 
The method used in creating the HRL, as well as the resulting framework is discussed 
below. 

B. CREATING THE HRL CONCEPT 

To establish the critical groundwork for developing the concept of the HRLs, 
multiple teleconferences and two separate five-day workshops took place at the 711th 
Human Performance Wing/Human Performance Integration Directorate of the Air Force 
Research Laboratory. The development team consisted of the thesis author and Dr. 
Hector Acosta, HSI professional and co-advisor of this thesis. 

Step I: Literature review 

The first step was to conduct an extensive literature review and document 
analysis. The literature reviewed is discussed in the previous chapter. From this review it 
was decided that the NATO Human View was the most appropriate framework on which 
to base the HRL. The reason for this was that it: a) provided a structured linkage from the 
engineering community to the technical domains of HSI; b) it established a logical and 
systematic framework for specifying the professional data collection and analysis actions 


31 



needed for effective HSI implementation; and c) it provided early coordination of task 
analysis efforts by both system engineering and HSI teams. 

Step 2: Establishing the underlying principles 

For the HRL framework to be accepted and useful to the HSI and acquisition 
communities, four underlying principles were used to guide the development of the HRL 
framework. 

1. Any risk management framework that is developed for HSI 
implementation should be consistent with existing risk management processes and, to the 
extent possible, complement existing program risk management metrics. 

2. HSI should define a process that supports the development and fielding of 
systems that minimize life cycle costs and result in human-centered designs that 
complement total system performance. This needs to be done in a way that is consistent 
with the realities of system engineering risk management and the need for rational, 
informed developmental tradeoffs. 

3. The model should facilitate vigorous HSI implementation early in the 
acquisition timeline to decrease risk of resulting life cycle cost penalties. 

4. In the context of a competitive cost, schedule, and performance-driven 
acquisition enterprise, any incremental expense to implement HSI must be identified in 
order to be funded. There currently is no process in DoD that identifies the incremental 
expenses involved with implementing HSI. 

Step 3: Development of HRL level definitions 

The decision was made to develop nine HRL levels to be consistent with the 
existing TRL structure. The reason for this decision was to encourage acceptance and use 
among the S&T and acquisition communities, particularly in developmental risk 
management structures. Each of the nine HRLs was given a preliminary definition. The 
following list provides the HRL definitions and a brief rationale for each: 
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1. Activation of Human Systems Integration: base-lining and commitment. 
At this level, the priority is to establish HSI infrastructure to support and manage initial 
planning and analysis efforts. 

2. Human Systems Integration analysis suite in support of component 
technology development. The goal of this second level is to ensure that substantial 
preliminary assessments and documentation has been accomplished. 

3. Component human touch point (i.e., human-system interaction, to include 
hardware and software) resolution: refining requirements thresholds. The primary 
objective of this level is to continue HSI analysis efforts across all domains, while 
identifying and communicating appropriate technological thresholds. 

4. Component human touch point engineering parameters and human 
performance indicators. The fourth level continues with iterative analyses and updates to 
HSI domain considerations bearing on the developing technology. 

5. Limited system human performance parameters/demonstration. In this 
level, HSI activities focus on supporting the increasing fidelity of breadboard technology. 

6. Field (relevant environment) validation of human performance prototypes. 
The goal of this sixth level is to make certain significant HSI evaluations are conducted 
in relevant environments. 

7. Final Developmental Test & Evaluation/human performance parameters. 
The objective for this level is to ensure HSI involvement in system T&E. 

8. Operational Test & Evaluation/human performance parameters. This 
level focuses on accomplishing specific HSI analyses to support OT&E events. 

9. Sustainment: Initiation of capability gap feedback cycle. The goal of the 
ninth and final level is to begin iterative review and verification of fielded systems to 
support ongoing sustainment. 

Step 4: Development of more detailed level descriptions 

The definitions developed in the previous stage established the framework to 

allow for the more detailed development of the HREs. This was accomplished by 
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elaborating the products of the Human View and explicitly developing their linkage to 
the concept of an HRL. There were five sub-steps carried out at this stage of the 
development process: 

1. An exhaustive list of HSI goals and considerations was developed. 

2. The HSI goals and considerations were arranged sequentially based upon the 
location in the acquisition life cycle. 

3. The unique and/or the more often iterative activities required to collect HSI 
data were identified. 

4. Data types were translated into language across the HSI domains, targeting 
clearly specified requirements and acquisition documents as they evolve (with the 
understanding that all data converges toward specifications). 

5. Data collection activities were linked with the standardized professional 
activities and process titles (e.g., Initial HSI Assessment, Hazard Analyses, Cognitive 
Task Analyses) that need to be executed by HSI practitioners in the collection of valid, 
reliable, and accurate HSI data. 

Step 5: Synthesis of the HRLs with the Human View 

The output from step 4 of the HRL development was synthesized with the Human 
View, cross-referenced with the HSI Integrated Framework Version 1.3, and entered into 
a Microsoft Excel Workbook. The HRL framework is summarized in Table 8, with the 
much more detailed spreadsheet included in Appendix F. 

Table 8. Human Readiness Level Framework Overview 


Human Readiness Level 

Description 

1. Activation of Human Systems 

Integration: base-lining and commitment. 

Lowest level of socio-technical readiness. 
HSI infrastructure is setup within Systems 
Engineering planning. Total system 
analysis from both functional relationship 
and organizational perspectives occurs. 
Activity examples include front-end 
analyses, preliminary functional allocation, 
and initial HSI assessment and plan. 
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2. Human Systems Integration analysis 
suite in support of eomponent teehnology 
development. 

Signifieant HSI input to aequisition 
development and doeumentation oeeurs. 
Aetivity examples inelude initial human- 
maehine interfaee assessment, generation 
of Target Audienee Deseription, and 
threat/hazard assessment. 

3. Component human toueh point 
resolution (i.e., human-system interaetion, 
to inelude hardware and software): 
refining requirements thresholds. 

Multiple needs analyses and studies are 
eondueted in support of requirement 
definitions. HSI domain assessments 
inform ongoing development aetions, as 
well. Aetivity examples inelude human-in- 
the-loop analyses, sub-system hazard 
analysis, and HSI plan update and revision. 

4. Component human toueh point 
engineering parameters and human 
performanee indieators. 

Iterative evaluation and analysis of eaeh 
HSI domain takes plaee and provides 
eritieal items of eonsideration bearing on 
system design. Aetivity examples inelude 
usability testing, development of human- 
eentered souree seleetion, and updating the 
human-maehine interfaee assessment. 

5. Limited system human performanee 
parameters/demonstration. 

Various HSI assessments and testing are 
performed to support the signifieant 
inerease in fidelity of breadboard 
teehnology. This ineludes supporting “high 
fidelity” laboratory integration of 

eomponents. Aetivity examples inelude 
examining safety and oeeupational health 
design features and eognitive task analyses. 

6. Field (relevant environment) validation 
of human performanee prototypes. 

Representative model or prototype system 
is tested in a relevant environment. 
Evaluations of human performanee 
embedded in demonstration system oeeur 
and HSI predietive models are updated. 
Aetivity examples inelude testing human 
reliability and usability of prototype in a 
high-fidelity laboratory environment or in 
simulated operational environment. 
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7. Final Developmental Test & 
Evaluation/human performance parameters. 

Significant HSI participation in system test 
events occurs. Iterative evaluation and 
analysis of each HSI domain continues as 
well. Activity examples include error and 
fault analysis to cover human error 
performance, equipment operability, safety 
procedures, and error recovery 

mechanisms. 

8. Operational Test & Evaluation/human 
performance parameters. 

Special human-centric analyses are 
conducted to update thresholds, objectives, 
and evolving criteria for Operational Test 
& Evaluation. Iterative evaluation and 
analysis of each HSI domain continues as 
well. Activity examples include system 
hazard analysis and HSI domain tradeoff 
studies. 

9. Sustainment: Initiation of capability gap 
feedback cycle. 

Extensive and iterative review and 
verification of fielded system begins, as 
well as post-product improvement 
evaluations for the next incremental builds. 
Activity examples include post-fielding 
training evaluation analysis and sustaining 
a hazard analysis for the fielded system. 


The HSI Integrated Framework Version 1.3 was developed by the U.S. Navy’s 
Space and Naval Warfare Systems Command to provide specific guidance on how to 
integrate HSI processes, products, and tools into Defense Acquisition (Risser, Belk, 
Smillie, Gepp, 2009). The HSI Integrated Framework Version 1.3 was used specifically 
to expand the HRL framework with any relevant detail not previously included. This 
allowed the final decisions to be made regarding the HRL’s functional structure, 
processes, and intent. 

C. LINKING THE HRL FRAMEWORK WITHIN THE ACQUISITION LIFE 
CYCLE 

The intent was for each HRL to provide a risk management metric that follows a 
scale from 1 (HSI base-lining & commitment) to 9 {capability gap feedback cycle). Risk, 
for purposes of HRLs, is mitigated by professional analytical and managerial activities 
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that link critical items of consideration to domain-speeifie design deeisions, ineluding 
well-informed tradeoffs. A program determined to be at HRL-1 is at the lowest level of 
HSI readiness (thus, highest level of HSI risk), where the initial aetivation of HSI 
eommitment and proeesses oeeurs. By the time the program has reaehed HRL-9, it has 
progressed through signifieant HSI analyses, requirement threshold refinements, field 
validations of human performance prototypes, and extensive developmental and 
operational Test and Evaluation (T&E) of human performanee parameters. Ultimately, a 
program’s human-centered maturity is aehieved through the performanee of HSI 
aetivities which address the consideration of end-users and other stakeholders in the 
speeification, development, and operation of a system. 

Eaeh HRE level provides a deseription of the type of aetions required to take the 
next step towards inereasing the socio-technical maturity of a program with respect to 
those milestone-sensitive phases in a teehnology’s life eyele. Eor instance, the designated 
eriterion threshold to be met at Milestone A in the Defense Aequisition Eife Cycle 
Management System is HRE-2; HRE-6 should be achieved prior to Milestone B; and 
HRE-8 prior to Milestone C. Eigure 3 illustrates the HRE milestone progression below 
(the milestones are also discussed in more detail in Appendix C). 
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By structuring appropriate sequencing of HSI activities and tracking the progress 
of HSI planning and exeeution, the HRL distinguishes itself as a signifieant risk 
management tool for proeess owners and decision makers in Defense Acquisition. Unlike 
TRLs where teehnology maturity descriptors serve as the basis for level assignment, the 
HRL functions by basing its HSI-related risk and maturity level off of the exeeution and 
outcome of those milestone-sensitive management and analytical activities that have been 
designated in eaeh progressive HRL. Higher HRL levels are intended to be consistent 
with more complete specification, including the effects of tradeoff decisions. This also 
serves to differentiate the HRL from the Human View. For instance, the Human View has 
been tailored to arehitectural models and serves as a repository for required HSI data. It 
does not, however, provide the data. It is only by doing the unspecified and carefully 
crafted data collection and analyses delineated within tools such as the HRL that HSI data 
ean be produced. 

D. SUMMARY 

This ehapter described the proeess by whieh the Human Readiness Level was 
developed. The next chapter builds upon these efforts by describing a subject matter 
expert (SME) evaluation of the HRL. The focus of this initial evaluation will be on the 
HRL framework’s overall worth and usability. 
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IV. HUMAN READINESS LEVEL EVALUATION 


A. BACKGROUND 

In order for the HRL to be incorporated within DoD as a legitimate HSI planning 
tool and complementary (HSI-specific) metric in program risk management structures, it 
is necessary for it to first be evaluated and scrutinized by the community to which it has 
been developed for. This chapter describes an initial evaluation of the HRL framework by 
SMEs familiar with HSI and the DoD acquisition process. 

B. METHODOLOGY 

1. Instrument 

A questionnaire was designed to gain feedback as to the overall worth and 
potential (i.e., accuracy, ease of use, completeness) of the HRL framework described in 
the previous chapter. The questionnaire consisted of 51 items divided into five separate 
groups. The first four grouping of items were designed to obtain feedback on the HRL’s 
recommended milestone progression in the Defense Acquisition Life Cycle. Lor instance, 
HRL-2 is the criterion threshold to be met at Milestone A, therefore the first set of items 
pertained to HRLs 1 and 2. The last group of items concerned the proposed categories for 
future HRL framework efforts. Both the HRL framework and the evaluation instrument 
can be viewed in Appendix L and G respectively. 

Data were gathered from participants’ responses to the HRL review questionnaire 
based on their ratings of agreement from the five-point Likert scale ranging from 
“strongly agree” to “strongly disagree.” The corresponding responses to the statements 
were converted to a numerical value ranging from 1 to 5 (1 - “strongly agree” to 5 - 
“strongly disagree”). In addition, within all five sections, the participant had the 
opportunity to make open-ended comments. 


39 



2. Participants 


To obtain feedback from SMEs, the selection criteria for inclusion in the study 
was that the participants had to have experience and/or graduate level training in both 
HSI and Defense Acquisition. Consideration was given to focusing solely on the 
professionals working in these areas within the Air Force. This was due to the Air Force 
HSI community being accessible to the researcher. However, because of the limited 
expertise in this combined area, it was decided that the evaluation should be expanded to 
other services. Therefore, the questionnaire was distributed to 42 individuals within DoD, 
as well as one individual from the Canadian military. Because the SMEs served as 
participants to the research and were not subjects o/the research, the Naval Postgraduate 
School’s Institutional Review Board (IRB) determined the study protocol did not require 
IRB involvement. 

C. RESULTS 

Out of the 43 identified SMEs, a total of 15 HRE evaluations were returned (a 
35% response rate). Of those who responded, one-third represented SMEs from the Air 
Force (N=5), while the remaining two-thirds consisted of respondents from other 
branches of the military (Army=2, Navy=4, Marines=2, Coast Guard=l, Canadian 
military=l). As the HRE framework was developed by two Air Force individuals, it was 
important to examine whether there were differences in the levels of satisfaction with the 
system between Air Force, and non-Air Force respondents. Table 9 summarizes the mean 
(and standard deviation) of the levels of satisfaction for five groups of items. It can be 
seen that the level of satisfaction of both Air Force and non-Air Force affiliated 
respondents improved with each higher level of the HRE framework. 


40 



Table 9. Rating of Agreement Summary for the SME Evaluation of the HRE 

Eramework (1 - “Strongly Agree” to 5 - “Strongly Disagree”) 


SME Evaluation Report 

Affiliation 

HREs 1/2 

HREs 3/4/5/6 

HREs 7/8 

HRE-9 

Proposed 

Categories 

Air Eorce 

Mean 

3.04 

2.76 

2.49 

2.49 

2.16 

Std. Deviation 

0.66 

0.63 

0.54 

0.54 

0.21 

N 

5 

5 

5 

5 

5 

Other 

Mean 

2.33 

2.05 

1.97 

1.88 

1.89 

Std. Deviation 

0.48 

0.39 

0.37 

0.38 

0.33 

N 

10 

9 

9 

9 

8 

Total 

Mean 

2.57 

2.30 

2.15 

2.10 

1.99 

Std. Deviation 

0.63 

0.51 

0.48 

0.52 

0.31 

N 

15 

14 

14 

14 

13 


The Mann Whitney U test (the non-parametric equivalent of the between subjects 
t-test) was used to assess whether there was a significant difference between Air Eorce 
affiliated participants, and those from other services. This analysis was carried out by 
examining the mean level of satisfaction for the items corresponding to each of the five 
sections of statements. 

Although the non-Air Eorce affiliated respondents were more satisfied in each 
category, the difference between the two groups was only significant for the items 
associated with HREs 3 to 6 (Z= 2.0, p<.05). 
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The Friedman’s rank sign test (the nonparametric equivalent of the repeated 
measures Analysis of Variance) was used to assess whether there were significant 
differences in the satisfaction of participants with each of the four groups of HRLs. The 
test was found to be significant (chi square= 9.0, df= 3, p<.05). The respondents were 
significantly more satisfied with the higher HRLs than the lower HRLs. 

A summary of the open-ended comments made by the respondents is provided 
below (see Appendix I for complete list of all of the comments made). 

HRLs 1 and 2 

For the first section of statements regarding HRLs 1 and 2, many participants 
stated that both HRLs seemed reasonably aligned with the acquisition timeline and 
function. However, in regards to whether any critical information was missing, many 
commented that acquisition programs will likely require activities or information not 
necessarily found in this first version of the framework. Typical comments included: 

“These timings are probably program-dependent, but they seem reasonably 
aligned with the DODI 5000.02 timeline... ” 

“We don't know what we don't know. We may find that there are critical drivers 
that are not under our purview or within our awareness on any given program... ” 

HRLs 3, 4, 5, and 6 

Many comments made within this section cited a need for more detail and better 
defined descriptions. Typical comments made were: 

“Stating HRLs 3-5 all should occur “as soon as possible after MS A” does not 
help discriminate between the three. Timing for activity completion should be 
more defined. ” 

“Descriptors should better distinguish maturity levels, particularly HRLs 3 & 4.” 

“Reference to “component” in HRLs 3&4 can mislead that system level was 
bypassed or circumvented. ” 
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HRLs 7 and 8 


Of the various comments regarding HRLs 7 and 8, most were specific 
recommendations for improving different aspects of the HRL framework. These 
comments included: 

“Need to find a way to take findings from DT&E and inform / support OT&E test 
readiness evaluation / report. Almost all HRL testing can be accomplished 
unobtrusively in DT&E and in enough detail to know what to expect in OT&E. 
HRL determination in OT&E will have to be a natural outcome from the actual 
tests - it must be unobtrusive. ” 

“The analysis by CDRfor HRL-7 needs to be pretty comprehensive as the design 
is pretty much locked in after this point and changes become difficult to 
influence/implement. Appears activities for HRL-S should be better tailored 
toward what resulted from DT&E, and how will deficiencies be addressed prior 
to OT&E. ” 

HRL-9 

This section provided a limited amount of comments, and the comments did not 
appear to be centered around any particular theme. Example comments included: 

“Recommend this HRL/description be aligned with Post Implementation Review. ” 

“It is important to include feedback throughout the entire acquisition process, not 
just during HRL 9. ” 

Proposed Categories 

Each of the seven proposed categories contained comments that indicated relative 
satisfaction and agreement. Typical comments made for each proposed category include 
the following: 
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Acquisition & Sustainment Toolkit Linkages 


“Since Logistics considerations should, like manpower and training issues, be 
dealt with early, it is not unlikely that much lower HRL levels could be influenced 
bytheASTK.” 

Integration Notes and Comments 

“Agreed, additional details will be especially useful for individuals tasked with 
HSI that are not very familiar with it or with the acquisition process. ” 

Target Documents 

“Agreed, individuals can go look at those documents for additional detail on 
program. ” 

“Recommend considering list of target documents for NEXT phase here as well to 
highlight upcoming tasks for planning purposes. ” 

Aetion OPR 

“There should be a program office OPR, a Sponsor/MAJCOM OPR, and an HSI 
OPR. ” 

“Especially when individuals move on, it is nice to have a POC to go back to or 
identify that the work has been in process. ” 

Referenees 

“Will provide information to individuals performing the tasks on exactly what 
they need to do and where to go to find information. ” 

“Consider hyper linking or a separate reference database/dictionary. ” 

Produets of Aetivity 

“Strongly agree... for building off of effort in next HRL or activity ” 

Rough Cost Estimate 

“This provides PM with information on the cost of doing the HSI at each 
level/activity for a program. ” 
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“This will allow PM to pick and choose the most important activities based on 

budget restrictions. ” 

D. DISCUSSION 

In general, larger sample sizes permit greater confidenee in the results and ean 
increase the probability of uncovering a wide range of underlying issues being studied. 
However, as stated by Krosnick (1999): “recent research has shown that surveys with 
very low response rates can be more accurate than surveys with much higher response 
rate” (p. 540). The representativeness is more important than the sample size. Given that 
there is not a large number of individuals within DoD who are experienced in both 
acquisition and HSI, it was not possible to obtain a large number of responses from the 
‘right’ people. Valuable insight and key information was still gleaned from a large 
proportion of the community who will be using the HRL framework. The following 
paragraphs discuss the findings from the quantitative and qualitative data. 

HRLs 

The agreement ratings and comments regarding the four separate groups of HRLs 
indicated that the respondents were generally satisfied with the framework. However, a 
number of suggested changes were made. For the first group of HRLs 1 and 2, the typical 
concern was that the framework would likely require more activities or information to be 
truly complete. This was not unexpected, particularly because of the sheer amount of 
actions and activities needed to address all HSI domains. Although, the intent is to 
include an exhaustive list of HSI activities within the HRL framework, the list is 
understandably not yet complete in this first version of the framework. 

For the remaining three groups (HRLs 3/4/5/6, HRLs 7/8, and HRL 9), most 
comments made by the SMEs cited a need for more detail and better defined descriptions. 
Of particular importance was the timing of HSI activities contained within each separate 
HRL. The concern was that the schedule for activity completion was not always clear. 
For example, currently the HRL framework states that HRLs 3 through 5 should all occur 
“as soon as possible after MS A, prior to MS B.” This statement of timing, according to 
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one SME, did not help to discriminate between the three HRLs. Future improvements 
will need to focus on the explicit delineation of HRLs and their descriptions with regards 
to acquisition timing and milestone events. 

Other findings involved significant differences in the satisfaction of participants 
with each of the four groups of HRLs. Specifically, the Friedman’s rank sign test 
revealed that as the levels within the HRL framework increased, so did the respondents’ 
level of satisfaction. However, other than the ratings of agreement that were received, the 
open-ended comments did not reveal any distinctive reasoning for the SMEs’ preference 
of the higher HRLs. 

Taken at face value, their lower satisfaction of the earlier HRLs could simply 
indicate more effort and research should be placed on these levels. However, this would 
ignore other potential explanations worthy of consideration. For instance, Krosnick 
(1999) differentiates between two different strategies for responding to questionnaire 
items: optimizing and satisficing. Optimizing requires the respondent to put forth 
cognitive effort in order to generate the optimum answer. Conversely, when using a 
satisficing strategy, rather than expending the effort to generate optimal answers, 
respondents may compromise their standards and expend less energy. Satisficing is 
especially common when the respondents are required to answer a long series of 
questions on a wide range of topics (as with the HRL evaluation). Therefore, it is very 
possible that although the respondents were initially motivated to provide high-quality; 
optimized feedback at the beginning of the HRL evaluation, they may have become 
increasingly fatigued as the questionnaire progressed and shifted to a satisficing response 
strategy. 

Another interesting finding was that non-Air Force affiliated respondents were 
more satisfied in each category of the HRL framework (although the difference was only 
significant for the items in HRLs 3 to 6). Although the reason for this finding is difficult 
to ascertain without conducting post interviews, one reason could be that the HRL 
concept originated within the Air Force. Being more familiar with the idea of an HRL, 
the Air Force SMEs likely had a vested interest and were more critical in their responses. 
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Proposed Categories 

The final section of the HRL evaluation was focused on the eight proposed 
categories for future framework development efforts. All eight categories received many 
“strongly agree” and “agree” ratings from the SMEs. Along with several open-ended 
comments, the general feeling suggested that the proposed additions to the HRL 
framework is very relevant and will be useful once defined and fused into the HRL 
framework matrix. Of all of the proposed additions, the Rough Cost Estimate category 
received the most attention (i.e., number and length of comments). However, this was not 
surprising due to the cost-driven nature of most acquisition environments. 

E. SUMMARY 

The results of this evaluation suggested relative agreement among SMEs that this 
first iteration of the HRL concept, once fully developed, will serve as a valuable HSI 
planning tool and complementary (HSTspecific) metric in program risk management 
structures. In order to improve the framework, many SMEs recommended expanding the 
HRL’s list of activities to account for a more diverse variety of programs that exist in 
acquisition. Suggestions like these will be documented to form a basis of advocacy for 
future HRL development. 

This effort has examined the overall worth of the HRL; however it is only the first 
step in a continuing process of definition and validation. More professional scrutiny is 
needed. Therefore, the next chapter focuses on the validation of the HRL model when 
being practically applied in a current acquisition program. 
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V. A CASE STUDY OF THE USEFULNESS OF THE HRL 
FRAMEWORK AS APPLIED TO THE GROUND CONTROL 
STATION MODERNIZATION PROGRAM 


A. BACKGROUND 

In Chapter III, the initial SME evaluation of the HRL framework was described. 
However, that analysis was not applied to a particular acquisition program. In this 
chapter, a case study of the application of the initial HRL framework was carried out as 
used by a SME with reference to a specific acquisition program. Specifically, the HRL 
framework was evaluated to assess whether it contributes to the creation and sustainment 
of an acquisition program’s HSI Plan (HSIP). The program chosen for this case study 
was the U.S. Air Lorce Ground Control Station Modernization (GMOD) program. 
Background regarding the GMOD and the HSIP will be presented, followed by the 
evaluation of the potential usefulness of the HRL in this particular context. 

1. GMOD 

The GMOD program is an acquisition currently being developed in the Air Lorce 
to advance the capabilities of legacy Ground Control Stations (GCSs). GCSs are the 
control centers that provide the facilities for human control of remotely piloted aircrafts 
(RPAs). The standard GCS consists of pilot and sensor operator workstations, as well as 
required support equipment. Currently, the GMOD program is using a phased capabilities 
development approach where each phase retrofits the legacy GCS with safety-critical, 
mission-critical, and reliability and maintainability capabilities. The ultimate goal for the 
GMOD program is to significantly improve human-machine interfaces and ergonomics to 
increase RPA mission capability. 

2. HSIP for GMOD 

The key to a successful HSI acquisition strategy for GMOD is comprehensive 
integration across the HSI domains and also across other core acquisition and engineering 
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processes. Ultimately, this integration is dependent on an accurate HSIP (DAU, 2009). 
The HSIP is the management tool used to plan, manage, and implement HSI in an 
acquisition program. It is the essential plan used in identifying HSI issues and 
recommending resolutions for obtaining the desired capability as identified by the 
program’s requirements and specifications (711 HPW/HPO, 2009). Employing an HSIP 
satisfies DoD Instruction 5000.02, where it calls for the program manager to have a plan 
for HSI in place early in the acquisition process to optimize total system performance, 
minimize total ownership costs, and ensure that the system is built to accommodate the 
characteristics of the user population. 

The HSIP serves as an evolutionary formal document that tracks HSI execution 
and identifies and mitigates HSI risk as the acquisition program progresses. For the 
present study, the usefulness of the HRL framework to directly support this effort will be 
evaluated. Consideration will be given of its ability to provide a potentially exhaustive 
list of candidate activities to be tailored for GMOD to populate HSI Planning to inform 
requirements, acquisition, and sustainment documents of record. In addition, the HRL 
will be evaluated in its ability to provide those using the HSIP a risk management metric 
and process that is similar to the TRL metric and process, but that explicitly links 
technology development to the effective design and specification of human-centered 
systems in Defense Acquisition. 

B. METHODOLOGY 

1. Instrument 

For the present study, a questionnaire was designed to gain feedback regarding 
the HRL’s ability to effectively contribute to the creation and sustainment of an 
acquisition program’s HSIP. Consideration was given to conducting a semi-structured 
interview. However, it was decided that, given the complexity of the HRL framework, a 
questionnaire would allow the SME to provide more thoughtful feedback, and allow him 
to carry out the evaluation at his convenience. The questionnaire contained a total of 16 
items separated into two sections as shown in Appendix H. The first section consisted of 
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six Likert Scale statements that focused on the completeness and relevance of the HSI 
activities and categories listed within the HRL framework. The second section consisted 
of 10 orders of merit ranking items that were designed to gain feedback as to the relative 
value/importance of the current and proposed categories contained in the HRL 
framework when being used to support an HSIP. Within both sections, the opportunity 
for comments was made available. 

2. Sample 

The questionnaire was administered to the chief HSI practitioner working within 
the GMOD acquisition program. As a senior research aerospace engineer and lead for 
HSI transition in the Air Force Human Performance Integration Directorate, this SME 
was recognized to have extensive knowledge and experience in developing and managing 
HSIPs. Much like the evaluation that took place in Chapter IV, the expert served as a 
participant to the research and was not subject of the research. Therefore, the Naval 
Postgraduate School’s Institutional Review Board determined the study protocol did not 
require IRB involvement 

C. RESULTS AND DISCUSSION 

In the first section of the HRL questionnaire, data were gathered based on the 
ratings of agreement from the five-point Likert scale ranging from “strongly agree” to 
“strongly disagree.” For the most part, there was a general agreement that the HRL could 
serve as a beneficial tool in the development of an acquisition program’s HSIP. The 
following statement represented the typical comments made by the case study SME: 

“Agreed... the HRL provides a context of how early in the program these 
activities should be conducted. Ties to risk by seeing when this should be done compared 
to when it is planned to be done (or that it hasn’t been done). ’’ 

Specifically, the expert felt the milestone-sensitive HSI activities listed in the 
HRE matrix could provide an effective framework for an HSI working group to tailor for 
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comprehensive HSI planning. Additionally, it was agreed that using the HRL framework 
as an HSI maturity metric can benefit HSI risk identification and management within the 
HSIP. 

The areas of disagreement were few, but important none the less. Consistent with 
findings from the larger review in the previous study, the expert did not feel the HRL 
framework contained an exhaustive list of HSI activities to account for all program needs. 
Specifically, the SME felt that more activities would be necessary to ultimately manage 
and sustain an HSIP throughout a program’s life cycle. The current activities were 
thought to be a good start; however, as this tool continues to mature and is used for a 
diverse variety of acquisition programs, more activities were felt to be needed. The 
following is the comment made by the SME regarding this issue: 

“7 think this is a great start, but just by the limited number of activities, I think 
that there are other activities that we will need to add as this tool is maturing, especially 
as we use this prototype tool for a diverse variety of acquisition programs. ” 

Other disagreement involved whether the proposed categories (i.e., column 
headings) for future HRE development efforts represented a complete list of the key areas 
of information needed to perform effective HSI Planning. A major category that was 
stated to be missing was some linkage to which HSI domain each of the sub-activities 
pertained to, or some other type of solution (e.g. color, numbering by domain, etc.) that 
would identify the domain. The domain links, as recommended by the expert, could 
introduce an even more valuable category that would calculate risk levels per domain. 

In the second section of the HRE validation, data collected were based on the 
order of value/importance for the current and proposed categories contained in the HRE 
framework when being used to support an HSIP (tied values of ranking were allowed). 
The HRL category that simply designated the HRE level (i.e., 1, 1.1, 1.2, etc.) was 
considered most important, followed by Products (of Activity), Activity, Sub-Activity 
Detail, Target Documents, References (tied with Target Documents), Action OPR, 
Integration Notes/Comments (tied with Action OPR), Rough Cost Estimate, and 
Acquisition & Sustainment Toolkit Linkages (did not receive a value ranking). The expert 
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stated that the framework categories that received higher rankings (particularly, HRL, 
Products (of Activity), and Activity) was due to their ability to efficiently tie into risk 
management structures and not only provide a metric on which to base developmental 
decisions, but they also provide program managers with visible indicators of what should 
be done compared to when it is planned to be done. Lastly, the category of Acquisition & 
Sustainment Toolkit Linkages was not ranked because the expert did not have enough 
familiarity with the toolkit to pass judgment of its importance. 

D. SUMMARY 

The results of this case study showed that the responses of a SME considering the 
usefulness of the HRL framework with reference to a particular acquisition program were 
consistent with findings from the study reported in the previous chapter. The SME 
thought the HRL could serve as a beneficial tool in the development of an acquisition 
program’s HSIP. The HSI practitioner who participated in this effort also considered the 
HRL to be a valuable HSI maturity metric that could benefit HSI risk identification and 
management within program risk management structures. In regards to needed 
improvements, the SME recommended to expand the HRL’s activities to account for the 
diverse variety of programs that exist in acquisition. In the next and final chapter, 
recommendations will be made on improvements that need to be made to the HRL 
framework. 
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VI. CONCLUSIONS AND RECOMMENDATIONS 


A. SUMMARY 

This thesis described the initial development of a Human Readiness Level 
framework, designed to complement the existing Technology Readiness Level currently 
being used within the Integrated Defense Acquisition, Technology, and Logistics 
(IDAT&L) Life Cycle Management System. Technology maturity assessment tools, such 
as the TRL, serve as systematic measurement systems that are used as entry and exit 
criteria for transitioning milestones and are integral components to program risk 
management structures. Yet, the TRL has proven incapable of consistently capturing the 
human-related aspects of technology development and their association with technology 
readiness. 

The proposed HRL framework was developed to add clarity to technical readiness 
assessments by emphasizing the socio-technical attributes of system development. 
Specifically, the HRL aims to reduce technology risks related to the human element by 
ensuring the adequate incorporation of Human Systems Integration during technology 
maturity evaluations and by explicitly linking technology development to the effective 
design and specification of human-centered systems in Defense Acquisition. The HRL 
accomplishes this risk reduction by providing a time and milestone-sensitive roadmap of 
activities that: a) detail critical organizational milestones (that ensure functional 
commitment to HSI); and b) define HSI domain-specific data collection and analysis, and 
a clear process for their management. The primary measure of HSI risk and maturity 
level is based on the execution and outcome of those milestone-sensitive management 
and analytical activities that have been designated in each progressive HRL. 

To evaluate the utility of the initial HRLs with the user population, two research 

efforts were carried out. In the first, an evaluation questionnaire designed to gain 

feedback as to the overall worth and potential usefulness of the HRL framework was 

given to 43 subject matter experts in the fields of HSI and Defense Acquisition. A total of 

15 responses were obtained. Results of this evaluation indicated relative agreement 
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among SMEs that the first iteration of the HRL concept, once fully developed, will serve 
as a valuable HSI planning tool and complementary (HSI-specific) metric in program risk 
management structures. In order to improve the framework, many SMEs recommended 
expanding the HRE’s list of activities to account for a more diverse variety of programs 
that exist in acquisition. 

The second research effort was a case study regarding the usefulness of the HRE 
framework when applied to a specific acquisition program. Specifically, the HRE was 
evaluated in its ability to effectively contribute to the creation and sustainment of the 
acquisition program’s HSI Plan for the Ground Control Station Modernization. Results of 
this SME evaluation suggested a general agreement that the HRE framework could serve 
as a beneficial tool in the development of an acquisition program’s comprehensive HSIP. 
Consistent with findings from the previous evaluation, the SME did not feel that the HRE 
framework contained all of the necessary HSI activities to account for all program needs. 
The feedback and lessons learned from both studies were documented and will form the 
basis for future HRE development efforts. Recommendations for improvements to the 
HRE framework are provided in the next section. 

B. RECOMMENDATIONS AND FURTHER RESEARCH 

Areas of particular importance that need to be addressed in the next iteration of 
the HRE framework are: 

1. Link HRLs to Cost Estimation Processes 

In the context of a competitive cost, schedule, and performance-driven acquisition 
enterprise, accurately funding HSI initiatives continues to be a critical component. Often, 
program requirements are inappropriately funded because the costs involving HSI- 
specific data collection and analysis were not estimated in advance. Such exclusions 
produce cost overruns, schedule delays, and deficiencies in performance that have 
considerable impact on the eventual total life cycle cost. Therefore, continued research 
must focus on linking HREs to acquisition cost estimation processes. Specifically, efforts 
should concentrate on detailing how HREs establish a basis for defining HSI-specific cost 
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estimation factors and elements paralleling Work Breakdown Structure links to system 
components. This will effectively “fund the requirement” by directly informing Planning, 
Programming, Budgeting, and Execution processes (employed by DoD for strategic 
planning and resource allocation). 

2. Standardize HRL Metrics within Technology Transition 

The DoD S&T community is tasked with ensuring that technologies are mature 
when DoD’s acquisition community takes over and integrates the technologies into 
weapon systems. This transition involves management and funding responsibilities to 
gradually shift from the lab to the product line. In order to support this transition with 
significant HSI influence, continued research should focus on connecting HRLs to 
Technology Transition Agreements. Specifically, efforts should define how HRLs 
articulate external dependencies on technology base projects and provide exit criteria for 
the technology to transition into the acquisition program (i.e., HSI metrics). This will 
provide S&T and acquisition program managers with demonstrated knowledge about the 
human-specific readiness and the potential risks of including the technology on a 
weapons program. 

3. Define HRL Assessment Procedures 

A standard repeatable method for determining the HRL achieved by a given 
technology must exist if the HRL is to be effectively used and consistently trusted within 
Defense Acquisition. Therefore, in a later stage of HRL development, a formal, 
systematic, and explicit process should be created for risk management and HRL 
reporting. The process will need to define how a measure is obtained and how to assign 
each HRL as an assessed program achievement value and risk management metric. 
Ultimately, this will be a vital step towards joining TRLs in program risk management 
structures within DoD. 
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C. CONCLUSION 

Although HSI is a fundamental component of a total systems approach, the 
successful integration of HSI into systems engineering and acquisition life cycles 
continues to be a challenge. Methodologies and tools, such as the HRL framework, are 
needed to enhance human/system performance, maximize operational effectiveness, and 
prevent cost overruns. It is recognized that further development and evaluation of the 
HRL concept is required. However, the initial framework presented in this thesis takes 
that first step towards providing acquisition professionals a comprehensive guide that 
ensures human-centric priorities are addressed throughout all phases and milestones of 
Defense Acquisition. 
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APPENDIX A. HARDWARE TRL DEEINITIONS, DESCRIPTIONS, 
AND SUPPORTING INEORMATION 


Hardware TRL Definitions, Deseriptions, and Supporting Information (DoD, 2009) 


TRL 

Definition 

Description 

Supporting information 

1 

Basic principles 
observed and 
reported. 

Lowest level of technology 
readiness. Scientific 
research begins to be 
translated into applied 
research and development 
(R&D). Examples might 
Include paper studies of a 
technology's basic 
properties. 

Published research that identifies the 
principles that underlie this technology. 
References to who, where, when. 

2 

Technology con¬ 
cept and/or appli¬ 
cation formulated. 

Invention begins. Once 
basic principles are 
observed, practical applica¬ 
tions can be invented. Appli¬ 
cations are speculative, and 
there may be no proof or 
detailed analysis to support 
the assumptions. Examples 
are limited to analytic 
studies. 

Publications or other references that out¬ 
line the application being considered and 
that provide analysis to support the 
concept. 

3 

Analytical and 
experimental criti¬ 
cal function and/or 
characteristic proof 
of concept. 

Active R&D is initiated. This 
includes analytical studies 
and laboratory studies to 
physically validate the 
analytical predictions of 
separate elements of the 
technology. Examples 
include components that are 
not yet integrated or 
representative. 

Results of laboratory tests performed to 
measure parameters of interest and com¬ 
parison to analytical predictions for critical 
subsystems. References to who, where, 
and when these tests and comparisons 
were performed. 

4 

Component and/or 
breadboard valida¬ 
tion in a laboratory 
environment. 

Basic technological compo¬ 
nents are integrated to 
establish that they will work 
together. This is relatively 
“low fidelity” compared with 
the eventual system. Exam¬ 
ples include integration of 
■'ad hoc" hardware in the 
laboratory. 

System concepts that have been consi¬ 
dered and results from testing laboratory- 
scale breadboard(s). References to who 
did this work and when. Provide an esti¬ 
mate of how breadboard hardware and 
test results differ from the expected sys¬ 
tem goals. 

5 

Component and/or 
breadboard valida¬ 
tion in a relevant 
environment. 

Fidelity of breadboard 
technology increases 
significantly. The basic 
technological components 
are integrated with reason¬ 
ably realistic supporting 
elements so they can be 
tested in a simulated envi¬ 
ronment. Examples include 
“high-fidelity” laboratory 
integration of components. 

Results from testing a laboratory bread¬ 
board system are integrated with other 
supporting elements in a simulated oper¬ 
ational environment. How does the “rele¬ 
vant environment” differ from the 
expected operational environment? How 
do the test results compare with expecta¬ 
tions? What problems, if any, were 
encountered? Was the breadboard sys¬ 
tem refined to more nearly match the 
expected system goals? 
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6 

System/subsystem 
model or prototype 
demonstration in a 
relevant 
environment. 

Representative model or 
prototype system, which is 
well beyond that of TRL 5. is 
tested in a relevant environ¬ 
ment. Represents a major 
step up in a technology’s 
demonstrated readiness. 
Examples include testing a 
prototype in a high-fidelity 
laboratory environment or in 
a simulated operational 
environment. 

Results from laboratory testing of a proto¬ 
type system that is near the desired con¬ 
figuration in terms of performance, weight, 
and volume. How did the test environment 
differ from the operational environment? 
Who performed the tests? How did the 
test compare with expectations? What 
problems, if any, were encountered? 

What are/were the plans, options, or 
actions to resolve problems before 
moving to the next level? 

7 

System prototype 
demonstration in 
an operational 
environment. 

Prototype near or at planned 
operational system. Repre¬ 
sents a major step up from 
TRL 6 by requiring demon¬ 
stration of an actual system 
prototype in an operational 
environment (e.g., in an air¬ 
craft. in a vehicle, or in 
space). 

Results from testing a prototype system in 
an operational environment. Who per¬ 
formed the tests? How did the test com¬ 
pare with expectations? What problems. 

If any, were encountered? What are/were 
the plans, options, or actions to resolve 
problems before moving to the next level? 

8 

Actual system 
completed and 
qualified through 
test and 
demonstration. 

Technology has been 
proven to work in its final 
form and under expected 
conditions. In almost all 
cases, this TRL represents 
the end of true system 
development. Examples 
include developmental test 
and evaluation (DT&E) of 
the system in its intended 
weapon system to deter¬ 
mine if it meets design 
specifications. 

Results of testing the system in its final 
configuration under the expected range of 
environmental conditions in which it will 
be expected to operate. Assessment of 
whether it will meet its operational 
requirements. What problems, if any, 
were encountered? What are/were the 
plans, options, or actions to resolve 
problems before finalizing the design? 

9 

Actual system 
proven through 
successful mission 
operations. 

Actual application of the 
technology in its final form 
and under mission condi¬ 
tions, such as those 
encountered in operational 
test and evaluation (OT&E). 
Examples include using the 
system under operational 
mission conditions. 

OT&E reports. 
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APPENDIX B. SOFTWARE TRL DEFINITIONS, DESCRIPTIONS, 
AND SUPPORTING INFORMATION 


Software TRL Definitions, Deseriptions, and Supporting Information (DoD, 2009) 


TRL 

Definition 

Description 

Supporting Information 

1 

Basic principles 
observed and 
reported. 

Lowest level of software 
technology readiness. A 
new software domain is 
being investigated by the 
basic research community. 
This level extends to the 
development of basic use, 
basic properties of software 
architecture, mathematical 
formulations, and general 
algorithms. 

Basic research activities, research 
articles, peer-reviewed white papers, 
point papers, early lab model of basic 
concept may be useful for substantiating 
the TRL. 

2 

Technology con¬ 
cept and/or appli¬ 
cation formulated. 

Once basic principles are 
observed, practical applica¬ 
tions can be invented. 
Applications are speculative, 
and there may be no proof 
or detailed analysis to sup¬ 
port the assumptions. 
Examples are limited to 
analytic studies using syn¬ 
thetic data. 

Applied research activities, analytic stu¬ 
dies, small code units, and papers com¬ 
paring competing technologies. 

3 

Analytical and 
experimental criti¬ 
cal function and/or 
characteristic proof 
of concept. 

Active R&D is initiated. The 
level at which scientific fea¬ 
sibility is demonstrated 
through analytical and labor¬ 
atory studies. This level 
extends to the development 
of limited functionality envi¬ 
ronments to validate critical 
properties and analytical 
predictions using non-inte- 
grated software components 
and partially representative 
data. 

Algorithms run on a surrogate processor 
in a laboratory environment, instrumented 
components operating in a laboratory 
environment, laboratory results showing 
validation of critical properties. 

4 

Module and/or 
subsystem valida¬ 
tion in a laboratory 
environment (i.e., 
software prototype 
development 
environment). 

Basic software components 
are integrated to establish 
that they will work together. 
They are relatively primitive 
with regard to efficiency and 
robustness compared with 
the eventual system. Archi¬ 
tecture development 
initiated to include interope¬ 
rability, reliability, maintain¬ 
ability. extensibility, 
scalability, and security 
issues. Emulation with cur¬ 
rent/legacy elements as 
appropriate. Prototypes 
developed to demonstrate 
different aspects of eventual 
system. 

Advanced technology development, 
stand-alone prototype solving a synthetic 
full-scale problem, or standalone proto¬ 
type processing fully representative data 
sets. 
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TRL 

Definition 

Description 

Supporting information 

5 

Module and/or 
subsystem valida¬ 
tion in a relevant 
environment. 

Level at which software 
technology is ready to start 
Integration with existing 
systems. The prototype 
implementations conform to 
target environment/inter¬ 
faces. Experiments with 
realistic problems. Simu¬ 
lated interfaces to existing 
systems. System software 
architecture established. 
Algorithms run on a proces- 
sor(s) with characteristics 
expected in the operational 
environment. 

System architecture diagram around 
technology element with critical perfor¬ 
mance requirements defined. Processor 
selection analysis, Simulation/Stimulation 
(Sim/Stim) Laooratory buildup plan. Soft¬ 
ware placed under configuration manage¬ 
ment. Commercial-of-the-shelf/ 
government-off-the-shelf (COTS/GOTS) 
components in the system software 
architecture are identified. 

6 

Module and/or 
subsystem valida¬ 
tion in a relevant 
end-to-end 
environment. 

Level at which the engi¬ 
neering feasibility of a 
software technology is 
demonstrated. This level 
extends to laboratory proto¬ 
type implementations on 
full-scale realistic problems 
in which the software 
technology is partially inte¬ 
grated with existing hard¬ 
ware/software systems. 

Results from laboratory testing of a proto¬ 
type package that is near the desired 
configuration in terms of performance, 
including physical, logical, data, and secu¬ 
rity interfaces. Comparisons between 
tested environment and operational envi¬ 
ronment analytically understood. Analysis 
and test measurements quantifying con¬ 
tribution to system-wide requirements 
such as throughput, scalability, and relia¬ 
bility. Analysis of human-computer (user 
environment) begun. 

7 

System prototype 
demonstration In 
an operational, 
high-fidelity 
environment. 

Level at which the program 
feasibility of a software 
technology is demonstrated. 
This level extends to opera¬ 
tional environment prototype 
implementations, where 
critical technical risk functio¬ 
nality is available for dem¬ 
onstration and a test in 
which the sof^vare technol¬ 
ogy is well integrated with 
operational hardware/soft¬ 
ware systems. 

Critical technological properties are 
measured against requirements in an 
operational environment. 

8 

Actual system 
completed and 
mission qualified 
through test and 
demonstration in 
an operational 
environment. 

Level at which a software 
technology is fully integrated 
with operational hardware 
and software systems. 
Software development 
documentation is complete. 

All functionality tested in 
simulated and operational 
scenarios. 

Published documentation and product 
technology refresh build schedule. Soft¬ 
ware resource reserve measured and 
tracked. 

9 

Actual system 
proven tnrough 
successful mission- 
proven operational 
capabilities. 

Level at which a software 
technology is readily 
repeatable and reusable. 

The software based on the 
technology is fully integrated 
with operational hardware/ 
software systems. All soft¬ 
ware documentation veri¬ 
fied. Successful operational 
experience. Sustaining 
software engineering sup¬ 
port in place. Actual system. 

Production configuration management 
reports. Technology integrated into a 
reuse “wizard." 
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APPENDIX C. MANAGEMENT OE TECHNOLOGY MATURITY IN 
THE DEEENSE ACQUISITION MANAGEMENT SYSTEM 


The Defense Acquisition Management System, as stated in the 2009 Defense 
Acquisition Guidebook (DAG), exists to manage the Nation's investments in technologies, 
programs, and product support necessary to achieve the National Security Strategy and 
support the United States Armed Forces. More specifically, it establishes a simplified and 
flexible management framework for translating capability needs and technology 
opportunities, based on approved capability needs, into stable, affordable, and well- 
managed acquisition programs that include weapon systems, services, and automated 
information systems (AISs) (DoDI 5000.02, 2008). 
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Figure 4. Integrated Defense Acquisition, Technology and Logistics Life Cycle 
Management System (After: DoDI 5000.02, 2008) 


Figure 4 depicts the evolution of technology in the DoD Acquisition Life Cycle. 
This process, beginning with User Needs and Technology Opportunities, uses Joint 
Concepts, integrated architectures, and an analysis of doctrine, organization, training, 
materiel, leadership and education, personnel, and facilities (DOTMLPF) to define 
needed capabilities and to guide the development of affordable systems through five 
subsequent phases: 
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1. Materiel Solution Analysis (MSA) 

2. Technology Development (TD) 

3. Engineering and Manufacturing Development (EMD) 

4. Production and Deployment (PD) 

5. Operations and Support (OS) 

Of the five phases, the first four place emphasis on technology and product 
development. During these early stages a new technology undergoes research and 
development, integration into systems, and finally the manufacture and distribution in 
weapon system form. The last phase. Operations and Support, attends to the system after 
it has been fielded and represents the majority of the actual time a system and its 
integrated technology must be managed (Nolte, 2008). 

In DoD acquisition, a crucial part of overall program management is the 
management of technology maturity and mitigation of technology integration risk. 
Therefore, per DoD Instruction 5000.02 Operation of the Defense Acquisition System, 
objective evaluations of technology maturity are required to be routine aspects 
throughout the entire acquisition life cycle. The evaluation, officially referred to by DoD 
as the Technology Readiness Assessment (TRA), is the formal, systematic, metrics-based 
process used to assess the maturity of critical hardware and software technologies to be 
used in DoD systems. The TRA is conducted by an Independent Review Team (IRT) of 
subject matter experts (SMEs) who use the TRE as the metric to assess technology 
maturity. 

All DoD acquisition programs going through the Defense Acquisition System 
must have a formal TRA completed at Milestone B, which is normally the formal 
initiation of an acquisition program, and also at Milestone C. However, assessments of 
technology readiness or TRA-like activities other than the formal TRAs at Milestones B 
and C take place over the acquisition life cycle. 

Using information and guidance found in DoD’s 2009 Technology Readiness 
Assessment Deskbook, the following paragraphs summarize how the knowledge 
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concerning technology maturity evolves over time within acquisition and discusses the 
mandated TRL levels that must be met to progress through the acquisition life cycle. 

A. EARLY EVALUATIONS OF TECHNOLOGY MATURITY 

In the Materiel Solution Analysis phase, an Analysis of Alternatives (AoA) is 
conducted to identify potential materiel solutions, based on a cost-benefit analysis. While 
this is occurring, early Systems Engineering (SE) activities, such as the proposed 
Engineering Analysis of Potential System Solutions, are conducted. The possible materiel 
solutions are then expected to undergo an Early Evaluation of Technological Maturity, 
provided sufficient technical information exists to support such an evaluation. This 
evaluation is an activity separate from the formal TRA and is conducted shortly before 
Milestone A. 

This body of work—the AoA, the early SE, and the Early Evaluation of 
Technological Maturity—forms the basis of the Technology Development Strategy 
(TDS) for evaluating the technology options in the materiel solution to the capability 
need identified in the approved Initial Capabilities Document (ICD). DoD relies on the 
TDS to show how the technologies (those known by Milestone A to be critical for the 
successful realization of the chosen materiel solution) will be demonstrated in a relevant 
environment (equivalent to TRE 6) before they are used in Engineering and 
Manufacturing Development. If the AoA and early SE work does not result in sufficient 
technical information to support adequate evaluation of technology maturity, acquisition 
programs are still expected to have an evaluation completed prior to Milestone A so that 
critical technologies can be matured during the subsequent Technology Development 
phase. 

The TRA Deskbook states a recommended best practice is to use the results of 
this early evaluation of technology maturity as follows: 

• To provide a basis for modifying the requirements if technological risks 
are too high. 
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• To support the development of Technology Maturity Plans (TMPs) that 
show how all likely critical technologies will be demonstrated in a 
relevant environment before preliminary design begins at the full system 
level. 

• To refine the TDS. 

• To inform the test and evaluation (T&E) community about technology 
maturity needs. 

• To ensure that all potential critical technologies are included in the 
program’s risk management database and plan. 

• To establish Technology Transition Agreements (TTAs) to articulate 
external dependencies on technology base projects and to define the 
specific technologies, technology demonstration events, and exit criteria 
for the technology to transition into the acquisition program. 

Ultimately, the early evaluation of technology maturity conducted at or shortly 
before Milestone A helps evaluate technology alternatives and risks and, by doing so, 
helps the acquisition Program Manager (PM) refine the plans for achieving mature 
technologies before Milestone B approval is sought. However, it is worth noting that this 
early evaluation of technology maturity is not a replacement for the Milestone B TRA 
(DoD, 2009). 

B. MILESTONE B TRA 

Throughout the beginning MSA and Technology Development phases, the DoD 
science and technology (S&T) community is tasked with ensuring that technologies are 
mature when DoD’s acquisition community takes over and integrates the technologies 
into weapon systems. This transition occurs at Milestone B which begins the EMD phase 
and, as mentioned earlier, normally marks the formal initiation of an acquisition program. 
To avoid programs from entering the EMD phase of the Defense Acquisition System 
with immature technologies. Title 10 United States Code (U.S.C.) Section 2366b 
requires, in part, that the Milestone Decision Authority (MDA) certify that the technology 
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in Major Defense Acquisition Programs (MDAPs), including space MDAPs, has been 
demonstrated in a relevant environment (TRL 6) before Milestone B approval. In 
addition, while 10 U.S.C. 2366b is only binding for MDAPs, the DoD is also requiring 
Major Automated Information System (MAIS) acquisitions to meet the same technology 
maturity standard at Milestone B. The formal TRA process and report serves as the basis 
of the MDA certification. 

In cases where the technology in the program has not been demonstrated in a 
relevant environment, the MDA is allowed to waive the certification requirement if it 
determines that such a requirement would hinder the DoD’s ability to meet critical 
national security objectives. However, as a matter of practice, such waivers will only be 
granted in extraordinary circumstances. In fact, DoDI 5000.02 requires Request for 
Proposal (RFP) language that prevents the award of an EMD contract if it includes 
technologies that have not been demonstrated to be mature. Therefore, a generic TRA not 
based on a planned technical solution is not acceptable by DoD. The TRA must be based 
on the technologies in the system and must be performed on all the competitors’ 
proposals in a source selection. 

C. MILESTONE C TRA 

Milestone C marks approval to enter Low-Rate Initial Production (TRIP) for 
hardware systems and limited deployment in support of operational testing for MAIS 
programs or for software-intensive systems that have no production components. TRL 7 
or higher is the expected state of technology maturity at Milestone C. 

The Milestone C TRA is conducted to reflect the resolution of any technology 
deficiencies that arose during EMD and it serves as a check that all critical technologies 
are maturing as planned. By the time an acquisition program reaches Milestone C, all 
critical technologies are expected to have appropriate advancements while continuing to 
mature through established TMPs. Any new technologies that have emerged should be 
identified, and their maturation plans reviewed. 
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For software, TRL 7 means that all source codes have been written and tested— 
not only as an independent module and/or component, but also as integrated into the 
whole system. The TRA at Milestone C is important for MAIS programs because it 

• Documents successful developmental test and evaluation (DT&E). 

• Examines plans for maintenance and upgrades to ensure that no new 
technologies are involved. 

• Determines whether algorithms will transfer successfully when host 
platforms are moved and full-scale applications are initiated in a real 
operational environment. 

• Identifies where new Milestone B reviews are needed for future releases to 
initiate efforts to improve performance and determines the architectural 
changes necessary to support these future releases. 
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APPENDIX D. IMPACT OF IMMATURE TECHNOLOGY IN 
DEFENSE ACQUISITION PROGRAMS 


In numerous reports to Congressional Committees, the Government 
Accountability Office (GAO) has addressed the problems with proceeding in system 
development with immature technologies and has detailed the resulting cost overruns, 
schedule delays, and performance shortfalls that have been undermining DoD’s buying 
power. According to the GAO, this dilemma is due in part to DoD’s difficulty 
transitioning technologies from a technology development environment to an acquisition 
program. Because the acquisition community frequently pulls technologies too early, it 
takes on the additional task of maturing the technologies—an activity that is the primary 
responsibility of technology developers—at the start of an acquisition program. They 
further state that the start of a program ushers in a high-cost, delivery-oriented phase in 
which the acquisition community is supposed to be focused on integrating subsystems 
and working on system development and demonstration. DoD has continued to allow the 
acquisition community to take over this task before the S&T community considers the 
technologies ready for transition (GAO, 2006). The lower the level of technology 
readiness, the more ground must be covered to bring the technology to the point at which 
it can meet the intended product’s cost, schedule, and performance requirements with 
little risk. Figure 5 illustrates this idea on the following page. 
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Figure 5. Using TRLs to Match Technology With Product Launch Requirements (From: 

GAO, 1999) 

The gap between the maturity of the technology and the product’s requirements 
represents the risks or unknowns about the technology. As each succeeding level of 
readiness is demonstrated, unknowns are replaced by knowledge and the gap becomes 
smaller. Ideally, the gap is closed before a new technology is included in a new product’s 
design (GAO, 1999). 

Large, highly visible acquisition programs are by no means immune to the 
problems surrounding immature technology. The Joint Strike Fighter (JSF), for example, 
is the most expensive aircraft program in DoD and has endured multiple acquisition 
schedule changes in order to reduce risk. The following GAO statement found in the 
2001 Joint Strike Fighter Acquisition report, illustrates how not meeting anticipated 
technology maturity can delay entry into key development phases: 

“Last year, we testified and reported that a key objective of the program’s 
acquisition strategy is affordability and that a part of that strategy — entering into 
engineering and manufacturing development with low technical risk—would not 
be achieved because technologies critical to meeting the program’s cost and 
requirement objectives were projected to be at low levels of technical maturity in 
April 2001, the date then scheduled for awarding the engineering and 
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manufacturing development contract. We stated that the program’s approach was 
not consistent with best practices in which technologies are more fully developed 
before proceeding into product development... Because of concerns about the 
adequacy of the Joint Strike Fighter’s short take-off and vertical landing flight 
test program, the maturity of its critical technologies, and other factors, the 
Fiscal Year 2001 National Defense Authorization Act directed that the contract 
for the aircraft’s engineering and manufacturing development not be awarded 
until certain criteria were met” (GAO, 2001, p. 1). 


The impact of immature technology can also be seen in the GAO’s 2006 review 
of 52 major DoD weapon programs. In this review, 90 percent of the programs were 
found to have started with immature technologies and more than half of the programs 
were working with immature technologies at design review, the time when DoD 
acquisition policy expects the design to be stable. By the time production began, one- 
third of the programs still did not have mature technologies. Not surprisingly, the GAO 
found that DoD research, development, test, and evaluation cost estimates increased 
dramatically for programs having immature technologies at program start (GAO, 2006). 
Figure 6 shows the average cost growth of DoD programs reviewed when technologies 
were mature and immature at program start. 


Maturity lovol 



Figure 6. Average Program Research, Development, Test, and Evaluation Cost Growth 
from First Full Estimate (sample of 52 DoD weapon programs) (From: GAO, 

2006) 


Programs that started with mature technologies averaged a fairly small 4.8 percent 


cost growth above the first full estimate, whereas programs that started with immature 
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technologies averaged about 35 percent cost growth. This includes some programs that 
experienced significantly greater cost growth. A consequence of this cost growth is that 
DoD typically delivers weapon systems late, reduces quantities to stay within cost 
estimates, shifts funds away from other projects to make up for added costs, or some 
combination of the three (GAO, 2006). 
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APPENDIX E. THE NATO HUMAN VIEW 


All of the information contained in this appendix was derived from the NATO 
Human View Handbook, which was prepared by the NATO RTO HFM-155 Human 
View Workshop. The purpose of a human view is to capture the human requirements and 
to inform on how humans interact with systems. The Workshop panel’s final set of 
products that composes the NATO Human View is listed below: 

HV-A Concept 

HV-B Constraints 

HV-C Tasks 

HV-D Roles 

HV-E Human Network 

HV-F Training 

HV-G Metrics 

HV-H Human Dynamics 


HV-A CONCEPT 

The HV-A is a conceptual, high-level representation of the human component of 
the enterprise architecture framework. Its purpose is to visualize and facilitate 
understanding of the human dimension in relation to operational demands and system 
components. It serves as both the single point of reference and departure to depict how 
the human will impact performance (mission success, survivability, supportability, and 
cost) and how the human will be impacted by the system design and operational context 
(personnel availability, skill demands, training requirements, workload and well-being). 
The HV-A has a close relationship with other architecture products that provide high- 
level concepts. 
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The elements of the HV-A may include: 

• Pictorial depictions of the system and its human component 

• High level indicators of where human system interactions may occur 

• A textual description of the overall human component of the system 

• A use case which describes the human process 

HV-B CONSTRAINTS 

Not only is the human the most important and unique system within the system- 
of-systems, but it can also be the weakest link or highest risk in that system. Therefore 
expressing the capabilities and limitations of the human in the system is required. The 
HV-B contains the set of data elements that are used to adjust the expected roles and 
tasks. It acts as a repository for different sets of constraints that may affect parameters of 
different views that may impact the human system. If a system requires a human 
interface, then the system must be designed to accommodate the human in such a way as 
to account for the human limitations, and to support/maintain the human to at least a 
minimum acceptable level. 

Due to the range of information captured in the HV-B, six sub-products capturing 
specific subsets of data have been defined for the HV-B. These are broken into two 
subsets. Personnel, containing four sub products, and Human Factors, containing two sub 
products. 

Personnel Sub Products: information about personnel available to participate in the 
roles: 

• Manpower Projections (HV-Bl) - illustrates predicted manpower 
requirements for supporting present and future projects that contribute to 
larger capabilities 

o Understand manpower forecasting to allow initial adjustments in 
training, recruiting, professional development, assignment and 
personnel management 
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o Anticipate impacts (and timeframe) related to number(s) of 
personnel, personnel mix, Military Occupational Structure 
Identification (MOSIDs), Rank/level distribution, and, 
postings/relocation(s) of personnel 

o Ensure sufficient number of personnel with necessary Knowledge, 
Skills, Abilities (KSAs) are ‘ready and able’ to support fielding of 
future program 

Career Progression (HV-B2) - illustrates career progression as well as the 
essential tasks, skills, and knowledge (and proficiency level) required for a 
given job 

o Address impacts of alternative system and capability designs on 
career progression; 

o Determine jobs available given an individual’s current job and 
occupation; 

o Assess competencies required for each individual job; and 

o Support personnel planning by identifying availability of 
individuals with necessary competencies early in acquisition 
process 

Establishment Inventory (HV-B3) - Defines current number of personnel 
by rank and job within each establishment 

o Supports forecasting of trained effective strength 

o Supports predicting number of people that must be trained, 
recruited, etc. to fill gaps required for ‘out years’ 

Personnel Policy (HV-B4) 

o Defines the various department policies dealing with (governing) 
HR issues 



o Ensures that personnel are fairly considered, properly treated, well 
looked after and supported in a legal, moral and ethical manner 
while employed with the department 

o HR documents, such as policies, doctrine, laws, benefits, pay, 
SOPs, etc. 

Human Factors Sub Products - data related to the capabilities of the humans 
assigned to roles: 

• Health Hazards (HV-B5) 

o Considers the design features and operating characteristics of a 
system that can create significant risks of illness, injury or death 

o Aims to eliminate minimize or control both short- and long-term 
hazards to health that occur as a result of system operation, 
maintenance and support 

o Hazards may include system, environmental or task hazard 
assessment; air quality control assessment; noise/vibration 
pollution evaluation; impact force, shock protection; WHIMS 
evaluation of tasks; radiation/LASER protection; CB protection; 
extremes of temperature, etc. 

o It may include aspects of survivability, i.e., limiting the probability 
of personal injury, disability or death of personnel in their 
interactions with the system. This can include providing protection 
from attack, and reducing detectability, fratricide, system damage, 
personnel injury and cognitive and physical fatigue 

• Human Characteristics (HV-B6) 

o Considers the physical characteristics of an operator and 
movement capabilities and limitations of that operator under 
various operating conditions 
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o Aims to compare operator capabilities and limitations with system 
operating requirements under various conditions to match or 
eliminate operating capabilities 

o It may include aspects such as anthropometrical/medical data; 
reach data; range of motion data; physical strength data; visual and 
auditory assessment; speed or duration of activity data; cognitive 
workload; working memory capacity; ability to be security cleared; 
personality, motivation, etc. 

HV-C TASKS 

The HV-C describes the human-specific activities, i.e., the tasks that have been 
assigned to the humans in a system over its entire life cycle. It also considers how the 
functions are decomposed into tasks. (The term task in this product refers to a piece of 
work that can be assigned to a person). 

The HV-C may also: 

• Clarify the human-related functions in a system 

• Provide a justification for the allocation of functions between the humans 
and machines 

• Decompose these functions into a set of tasks that can be mapped to the 
roles identified in HV-D 

• Describe these tasks in terms of various criteria and the KSA requirements 

• Produce a task-role assignment matrix 

• Depict the inter-dependencies between different tasks, particularly across 
functional groupings 

• The information demands to perform specific tasks 

• The tools required to accomplish a task 

• Create interface design guideline on the basis of task requirements 
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The HV-C is very broad and can be used to capture all aspects of the human- 
related tasks in a system, including the allocation of tasks between humans and systems. 
This product is also closely related to the HV-D Roles, which will be described in the 
next section. There may be some overlap between the definition of tasks, roles and the 
assignment between them. More often, there may be multiple HV-C products 
representing different aspects of the human tasks in the architecture. In this case, the 
multiple products can be labeled consecutively within the HV-C context. 

HV-D ROLES 

The HV-D describes the roles that have been defined for the humans interacting 
with the system. A role represents a job function defining specific behavior within the 
context of an organization, with some associated semantics regarding the authority and 
responsibility conferred to the user in the role, and competencies required to do the job. 
The role structure can be mapped to the human task decomposition to define the 
organizational responsibilities, and relationships between roles can be defined which 
provides the basis of the organizational structure. 

The HV-D may define additional attributes of a role including: 

• Responsibility - a form of accountability and commitment; roles are 
generally defined by their responsibilities 

• Authority - is the access ability of an individual user to perform a specific 
task 

• Competencies - the quality of being able to perform; a combination of 
knowledge, skills and attributes; these should be trainable and measurable 

• Multiplicity - a role may be performed by a human user or by many 
human users at the same time 

The HV-D is closely related to the HV-C, as the identified tasks need to be 
allocated to roles, and competencies for the roles defined based on the assigned tasks. 
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The HV-D can also be extended to include a “concept of work” to describe the 
distribution of responsibilities among humans and specific requirements for those 
responsibilities. 

HV-E HUMAN NETWORK 

The HV-E captures the human to human communication patterns that occur as a 
result of ad hoc or deliberate team formation, especially teams distributed across space 
and time. 

Elements of the HV-E may include: 

• Role groupings or teams formed, including the physical proximity of the 
roles and virtual roles included for specific team tasks 

• Type of interaction - i.e., collaborate, coordinate, supervise, etc. 

• Team cohesiveness indicators - i.e., trust, sharing, etc. 

• Team performance impacts - i.e., synchronization (battle rhythm), level of 
engagement (command directed) 

• Team dependencies - i.e., frequency/degree of interaction between roles 

• Communication/Technology impact to the team network - i.e., distributed 
cognition, shared awareness, common operational picture, etc. 

HV-F TRAINING 

HV-E is a detailed accounting of how training requirements, strategy, and 
implementation will impact the human. It illustrates the instruction or education and on- 
the-job or unit training required to provide personnel their essential tasks, skills, and 
knowledge to meet the job requirements. This view can also address the development of 
additional training programs to meet the requirements of new capabilities. 

Data elements of the HV-E may include: 

• As-is training resources, availability, and suitability 
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• Risk imposed by to-be operational and system demands 

• Cost and maturity of training options for tradeoff analysis 

• Address impacts of alternative system and capability designs on training 
requirements and curriculums 

• Determine training required to obtain necessary knowledge, skills, and 
ability to support career progression 

• Differentiation of basic, intermediate, or advance job training; operational 
vs. system specific training; and individual vs. team training 

HV-G METRICS 

The HV-G can be its own product or incorporated into another architecture metric 
view, such as the SV-7 (in MoDAF and DoDAF). It provides a repository for human- 
related values, priorities and performance criteria, and maps human factors metrics to any 
other human view elements. It may map high-level (qualitative) values to quantifiable 
performance metrics and assessment targets or it may map measurable metrics to human 
functions, i.e., human performance specifications. It provides the basis for any human 
factors assessments to underpin enterprise performance assessments and the foundation 
for requirements tracking and certification. For example, it may include task standards as 
well as performance measures. 

Elements of HV-G may include: 

• Human Factors Value definitions level 1.. .n 

• Human Performance Metrics (what is to be measured) 

• Target Values (what quantifiable value is acceptable) 

• Key Performance Parameters 

• Human Task to Metrics mapping 

• Value to design element mapping 

• Methods of compliance 
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HV-H HUMAN DYNAMICS 


The HV-H captures dynamic aspects of human system components defined in 
other views. These are dynamic aspects in the sense that states, conditions, or 
performance parameters may change over time, or as a result of triggering events. It pulls 
together definitions from across the Human View to be able to communicate enterprise 
behavior. It provides inputs to human behavior and executable models that may be 
supported by simulation tools. There are many different human models and simulations 
that can be used to develop dynamic models; this view can provide stimuli and design 
aspects for these models. 

Features of the HV-H may include: 

• States (e.g. snapshots) and State Changes, e.g. 

o Organizational/team structure 
o Task/Role assignments to people 
o Team interaction modes 

o Demands on collaboration load (e.g. need to spend effort in 
building shared awareness, consensus-finding, communicating) 

o Task switches/interruptions 

• Conditions (e.g. triggering events or situations; scenarios) 

o Critical / frequent / representative / typical scenarios 
o Operational constraints (e.g. extensive heat stress) 
o Time conditions: sequence, duration, concurrency 

• Performance outcomes (observed or predicted), e.g. 

o Workload 
o Decision speed 
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o Team interaction/collaboration style 
o Trust in commanders intent 

o Quality of shared awareness/coordination/implicit communication 
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APPENDIX F. HUMAN READINESS LEVEL FRAMEWORK 


The following pages contain the first iteration of the Human Readiness Level 
framework. The contents of the framework drew heavily upon the technical details of the 
NATO Human View Handbook and the Navy’s HSI Integrated Framework Version 1.3 
(NATO RTO HFM-155 Human View Workshop, n.d.; Risser, Belk, Smillie, Gepp, 
2009). At this point in the HRL’s evolution, only the first three columns have been fully 
populated. The remaining columns represent the proposed categories of future HRL 
developmental efforts. 
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HRL 

Activity 

Sub-Activity Detaii 

Acquisition 
&Sustainment 
Tooikit Linkages 

integration 
Notes and 
Comments 

Target 
Documents 
(e.g. iCD or 
RFP) 

Action OPR 
(generaiiy 
within HSi 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL1 = 

Activation of 
Human Systems 
Integration 
Process: 
Baselining & 
Commitment 

HRL-1 activities should 

occur as soon as 
possible prior to 
Milestone A (MS A) 









1.1 

Program HSI commitment 
actions; setting up HSi 
infrastructure within SE 
planning 

1.1.1 Appoint HSI Lead, 

1.1.2 Initiate Program HSI Plan (HSIP) draft 
including HSI Risk mitigation process 

1.1.3 Define process for continuously rolling up 

HSIP milestones into SEMP at proper level of 
resolution 

1.1.4 Synchronize with SMART, and 

1.1.5 Establish reiationship/MOAwith primary 
service HSI support unit 








1.2 

Total system analysis from both 
functional relationship and 
organizational perspectives 
with an empahsis on system 
boundaries 

1.2.1 Whole system Front End Analysis (System, 
System of Systems (SOS)and Family of 

Systems(FOS) Definition); 

1.2.2 Mission Scope Analysis 








1.3 

HSI Assessment for HRL1 (HSIA 
1): Assessment based on best 
available solution description 
(in CBA, AoA, CONOPs, ICD, 
etc.) acrossall HSI domains to 
define level of risk in each. 

First priority goal is to affect 
design; second priority is to 
inform sustainment 

1.3.1 Manpower /Workload/ Functional Allocation 

1.3.2 Personnel / KSA clusters/ Selection/ 

Retention/ Career Path 

1.3.3 Training/KSAs 

1.3.4 Environment 

1.3.5 Survivability / Situation Awareness Force 
Protection/ Egress systems 

1.3.6 Habitabiltly / Sustained Op^ Facilities 

1.3.7 Occupational Health 

1.3.8 Safety/ HFACs 

1.3.9 Human Factors Engineering 

1.3.10 Establish Initial issues log with mitigation 
plans 

1.3.11 Discussand define HSI risk matrix reporting 
process and criteria for PM review and approval 


Provides baseline for all 
HSI-related activities. All 
domains are 
addressed.// 

Eaoh domain diotates 
oritloal Items of 
consideration bearing on 
system design 
depending on the 
(Initially likely, and later 
definite) Human Touch 
Points (HTPs) in a 
system as it 6\ol'«s. 
Risk, for purposes of 
HRLs, is mitigated by 
Professional analytical 
and managerial actlutles 
that link critical items of 
consideration to domain- 
specific design 
decisions, Including well 
informed trade-offs. 
Criclcai Initial step In 

HSI planning. 
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1.4 


Use this as planning tool for the 
HSIWG to ensure maximum 
coordination among data- 1.4.1 
collection and analysis 
activities 


Integrated activity worksheet for HRL1 


1.5 


Systematic prediction of errors 
that could be made by the 
humans in the system 


|l.5.1 Human Reliability Assessment (HRA) 


1.6 


Initial HFE analysis of projected 
system functions, identifying 
expected reliability based on 
human performance vs 
automation. Extract lessons 
learned to feed back to 
capability requirements and 
feed forward to desgn 
implications 


1.6.1 Identify HFE high drivers and lessons learned 
from legacy systems 

1.6.2 Identify user needsand environmental 
constraints on human performance 

1.6.3 Contribute to assessment and development of 
system concepts/ materiel solutions 

1.6.4 Assist in development of the cost estimate 
based on anticipated Manpower 

1.6.5 Conduct Mission Task Analysis 

1.6.5.1 Develop Mission Scenarios 

1.6.5.2 Conduct Misson Level Functional Analyse 

1.6.5.3 Develop Design Reference Scenarios 

1.6.5.4 Prelimary Function Allocation. 

1.6.5.5 Conduct Mission Level Task Analysis 



Woiksheets should be 
used prior to detailing in 
HSIP. Several of the 

data collection activities 
are listed as rows below 
but others willl have to 
be "planned" based on 
specific proram needs. 

A critical goal should be 
to minimize interference 
with operational SMEs 
(minimize number of 
visits by different data 
collection teams when 
overlapping data are 
sought). 
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1.7 

Systematic prediction of 
conditions inherent in normai 
operations (inciuding combat) 
that might iead to death, 
disabiiity, iiiness, injury, or 
performance deterioration. 
Shouid inciude a system safety 
engineering anaiysisfiie to 
document system design 
mitigations as hazards are 
identified 

1.7.1 Hazard Analysis / preliminary system, safety 
analyses / Identify safety issues, lessons learned and 
drivers from legacy systems 

1.7.2 Contribute to assessment and development of 
system concepts and materiel solutions 

1.7.3 Develop Preliminary Hazard List (PHL) for 
each system concept and materiel solution 

1.7.4 Assist in development of cost estimates based 
on anticipated safety activities & contributions for 
proposed concept^solutions 

1.7.5 Participate in Mission /Task Analysis 
[Contribute to TDRA while identifying drivers and 
requirements] 

1.7.6 If munition, perform Threat Hazard Assessment 
(THA): 

1.7.7 Generate PHL 








1.8 

Identify manning needs, 
manpower lessons learned, 
and high drivers from legacy 
systems 

1.8.1 Establi^ manpowr thresholds and objectives 

1.8.2 Establi^ level and duration criterion for high 
sustained workload. 

1.8.3 Identify manpower goals and parameters 
(DoD1100.4) 

1.8.4 Define the human performance characteristics 
of the user population (based on misaon and system 
requirements, target occupational specialties, 
recruitment and retention trends) 

1.8.5 Contribute to assessment and development of 
system concepts / materiel solutions 

1.8.6 Assist in development of the cost estimate 
based on anticipated manpower 

1.8.7 Participate in Mission Task Analysis 
[Incorporate findings from Mission Task Analysis into 
initial TDRA) 
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1.9 


ISynchronize with DoDAF 


1.9.1 Coordinate with Requirements Manager (RM) 
on current state of OV-1 and CONORS; 

1.9.2 Begin specification of HV-A 

1.9.2.1 Review information and data components 
required to begin popuiating aii HVs 


1 . 10 ' 


Linear modeiing and studies 
/anaiyses of System Approach 
Alternatives 


1.10.1 Rian preiiminary iinear modeis 

1.10.2 Negotiate with stakeholders of "what if 
studies 

1.10.3 Begin coordination with Man in the Loop 
(MiL) simuiation experiments 


1.11a 


Rerform personnei aiternatives 
concept development and 
assessments 


1.11.1 Consider personnei aiternatives based on 
FSA (CJCSM /3170.01 C) 

1.11.1.2 Determine impacts on cost estimates 

1.11.1.3 Determine appropriate personnei 
approaches 

1.11.1.4 Assist in development of the cost estimate 
based on anticipated personnei activities 

1.11.2 Define the human performance 
characteristics of the user popuiation 
(based on mission and system requirements, 
target occupationai speciaities, and 
recruitment and retention trends) 


1.11b 


Rerform personnei aiternatives 
concept development and 
assessments 


1.11.3 Identify personnel shortfalls 

1.11.4 Use findings from the mission task analysis 
and other personnel assessments to identify KSAs 

1.11.5 Assess personnel readiness, personnel 
tempo, and funding issues that may impact the 
ability to fill personnel requirements 

1.11.6 Rerform an analysis of alternatives for 
personnel optimization 


OV-1; operational 
concept graphic. 
Concept (HV-A) 
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1.12.1 Perform a training effectiveness evaluation 
on legacy systems 

1.12.1.1 Provide recommendations to correct 
training deficiencies 

1.12.1.2 Identify training deficiencies and root 


Perform training alternatives 
concept development and 
assessments 


1.12.2 Assess human performance and effectiveness 
of training transfer 

1.12.3 Assist in development of the cost estimate 
based on anticipated training activities 

1.12.4 Participate in Mission Task Analysis 

1.12.4.1 Input mission scenarios to the training 
assessment 

1.12.4.2 Conduct assessments to determine KSAs, 
learning objectives, and training requirements for 
the proposed materiel solution 

1.12.4.2.1 Justify the training system 

1.12.4.2.2 Skillsand Training analyse 

1.12.4.2.3 Training Requirements Analysis 

1.12.4.2.4 Complete total training requirements 

1.12.4.2.5 Perform a media selection analysis 


Perform training alternatives 
concept development and 
assessments 


1.12.5 Perform analysis of alternatives for training 
optimization 

1.12.6 Technologie^cost trade-off 
analyses for training options 

1.12.6.1 Assist in development and evaluation of the 
training program planning of alternatives 

1.12.6.2 Consider tasks, conditions, and standards 
from the Functional Area Analysis 
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HRL 

Activity 

Sub-Activity Detaii 

Acquisition 
&Sustainment 
Tooikit Linkages 

integration 
Notes and 
Comments 

Target 
Documents 
(e.g. ICD or 
RFP) 

Action OPR 
(generally 
within HSi 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL2=HSI 

analysis Suite to 
support 
component 
technology 
development 

HRL-2 activities 
shouid occur as 
soon as possibie 
prior to MS A 









2.1 

HSI Assessment ~ HRL2 
(HSIA2). First priority goai is to 
affect design; second priority is 
to inform sustainment. 

2.1 Assess HSi considerations in each Domain 

2.1.1 Manpower/Workioad/ Functionai Aiiocation 

2.1.2 Personnel / KSA clusters/ Selection/ 

Retention/ Career Path 

2.1.3 Training/KSAs 

2.1.4 Environment/ 

2.1..5 Survivability / Situation Awareness Kll Chain 
Analysis Force Protection/ Egress systems 

2.1.6 Habitabiltiy / Sustained Ops/Facilities 

2.1.7 Occupational Health 

2.1.8 Safety/ HFACs 

2.1.9 Human Factors Engineering 

2.1.9.1 Define HFE Concept/Capability 

2.1.10 HSI WG review HSI Issues log and mitigaton 
progress 

2.1.11 HSI WG specifies and recommends risk 
matrix outcomes and HRL assignment to PM 


Re-Assessment based 
on best available 

Solution Description 
(Now to include solution 
alternative from AoA or 
Block definition and 
current state of 

CONORS, ICD, etc.) 
across all HSI domains 
to define level of risk in 
each. Each domain 
dictates critical items of 
consideration bearing on 
system design 
depending on the 
(initially likely, and later 
definite) HTPs in a 
system as it evolves. 
Risk, for purposes of 
HRLs, is mitigated by 
Professional analytical 
and managerial activities 
that link critical items of 
consideration to domain- 
specific design 
decisions, including well 
informed trade-offs. 
Provides baseline for all 
HSI-related activities. All 
Domains are addressed. 
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2.2 


Update/revise HSIP 


2.2.1 Using worksheet as described in HRL-1 to 
update sequence of activities remaining 






2.3 

Generate Target Audience 
Description and Airman 
Capabiltiy Document 

2.3.1 Use findings from the mission task analysis 
and other personnel assessments to identify KSAs / 
Human Performance requirements 

2.3.2 Analysis activities to define Manpower, 
Personnel and Training-related Human Performance 
parameters 

2.3.3 Assess personnel / readiness, personnel 
/tempo, and funding issues that may impact the 
ability to fill Personnel requirement^current and 
needed AFSCs/ hard to fill occupations / required 
KSAs 

2.3.4 Derive initial Personnel requirements'[New 
occupational specialties (unique skill sets)/hard to 
fill occupations Highly qualified personnel. 

Determine how requirements will be met 


2.4a 

Training Systems Requirements 
Analysis (TSRA)—The initial 
step in user requirements 
identification. It consists of 
mission/task analysis, training 
requirements identification, 
objectives'media analyse, and 
training systems basis analysis. 
A TSRA integrates the products 
of the Instructional System 
Development (ISD) process and 
the Systems Engineering (SE) 
process to describe the 

Training System to be 
procured. It serves as a 
required input to the System 
Training Plan. It is 
accomplished by the PM in 
coordination with the Lead 

Command and User Command 
(Aug 30, 2007 ... This TSRA 
process is consistent with DODD 
1430.13) 

2.4.1 Participate in Mission Task Analysis 

2.4.2 Input mission scenarios to the training 
assessment//JTA reviews 

2.4.3 Conduct assessments to determine KSAs, 
learning objectives, and /training requirements for 
the proposed materiel solution 

2.4.3.1 Justify the training system Learning 
Performance and Solution Analysis 

2.4.3.2 Skillsand Training Analysi^Training 
Requirements Analysis 

2.4.3.3 Complete total training requirements 

2.4.4 Perform a high level media selection analysis 

2.4.5.1 Perform an analyse of alternatives for 
training optimization 

2.4.5.2 New learning Technologies Cost trade-off 
analyses for training options 

2.4.5.3 Assist in development, and evaluation of the 
training / program planning 

2.4.5.4 Provide descriptions, methods, and 
technology requirements for the training concept 
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2.4b 


Training Systems Requirements 
Analysis (TSRA)—The initial 
step in user requirements 
identification. It consists of 
mission/task analysis, training 
requirements identification, 
objective^media analysis, and 
training systems basis analysis. 
A TSRA integrates the products 
of the Instructional 
SystemDevelopment (ISD) 
process and the Systems 
Engineering (SE) process to 
describe the Training System to 
be procured. It serves as a 
required input to the System 
Training Plan. It is 
accomplished by the PM in 
coordination with the Lead 
Command and User Command 
(Aug 30, 2007... This TSRA 
process is consistent with DODD 
1430.13) 


2.4.5.5 Consider tasks, conditions, and standards 
from the Functional Area Ana lysis (CJCSM /3170.01C) 

2.4.6 Educational analysis A process of reviewing 
the educational requirements, developing 
educational goals, and developing statements for 
achieving the goals 

2.4.7 Occupational analysis A process of identifying 
tasks that are closely related and grouped under an 
occupational specialty. 

2.4.8 Mission analysis A process of reviewing 
mission requirements and developing a mission 
statement, mission segments collective task 
statements and arranging the segmentsand tasksin 
a hierarchical relationship. 

2.4.9 Job analysis Identifies the jobs that define an 
occupational skill area and identifies duties and 
tasks that comprise each job 


2.5 


2.5.1 Complements 2.3, specialized HFE interview 
and validation process that translates operator/ 
Legacy Operator/Maintainer maintainer interpretations of mission activities into 
Vignettes process sequence descriptions that systems 

engineers can use as a basis for system specification 
and design 


2.6 


Generate Design Reference 
Missions 


2.6.1, Enabled by 2.5, Mission descriptions that in 
combination tap the full range of functional 
capabilities of the system 






2.7 


Perform Function Allocation 
analysis 


2.7.1 Much more informed version of work begun in 
1.6, allocating tasks to humansand machinesin a 
way that adds value to capability in term of cost, 
safety and performance. Exploits relative strengths 
of man vs. machine!! 


2.8 


2.9 


2.10 


Human Reliability Assessment 
(HRA) 


2.8.1 Based on predictions about compontent 
technologies, systematic prediction of errors that 
could be made by the humans in the system 


Update of Hazard Analysis, 
Threat/Hazard Assessment 
(THA): Preliminary Hazard 
Analysis (PHA) and 
Requirements Hazard Analysis 
(RHA) 


2.9.1 Systematic prediction of conditions inherent in 
normal operations {including combat) that might 
lead to death, disability, illness, injury, or 
performance deterioration. 

2.9.2 Derive initial SOH requirements' 

2.9.3 Update hazaards lists 


Initial Manpower Estimate 
Analysts; Generate TDRA (Top 
Down Requirements Analysis) 


2.10.1 Based on analyses to date, updated modeling 
and analysis against operational needs to generate 
manpower estimate working with A1 tools, 
representatives and process stakeholders (LOOM, 
etc.) 

2.10.2 Conduct initial TDRA [Compute time-on-task 
for readiness conditions and mission scenarios 

2.10.3 Produce data to support manpower 
determination from the allocated functions and tasks 
/ Initial Workload Analyas 

2.10.4 Perform an analysis of alternatives for 
manpower optimization [Identify capability gaps in 
manpower reduction. Quantify workload reductions 
for each specific alternative (AoA)] 

2.10.5 Derive initial Manpower requirements 
/Identify constraints and limitations 
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2.11 

Ergonomic^biomechanical / 
anthropometric assessment 

2.11.1 Preliminary analyas of notional human touch 
point (HTP) taking into account form, fit and function 
to accommodate both static and dynamic human at 
work; includes accessability and egress during 
emergency situations and normal work 

2.12 

Initial Human Machine 

Interface (HMI) assessment 

2.12.1 Preliminary analysis of interface design 
considerations primarily related to controls and 
displays but extending to any human touch point, 
with or without the use of tools and the design and 
diversity of the tools themselves 

2.13 

Review LCMP draft 

2.13.1 Identify stakeholders and update assumptions 
for current program state 

2.14 

Review TEMP draft 

2.14.1 Identify stakeholders and update assumptions 
for current program state 

2.15 

Activate risk management 
process 

2.15.1 Establish detailed administrative processes 
for project HSI issues log (in HSIP) 

2.16 

Syncronize with DoDAF and or 
incorporate/roll-up into SEMP 

2.16.1 Coordinate with RM on current state of OV-4 
and CONORS; begin specification of HV-B, D-1, and 
F-1. Review information and data components 
required to begin populating all HVs 
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2.17 

Consider high ROI use of mock- 
ups for phyacal layouts of HTPs 

2.17.1 Formally consider / plan generating mock- 
ups to allow high-level assessment of safety, 
communications, workflow, usability, and general 
form and fit of design alternatives These are 
especially useful when experienced legacy system 
operators and maintainersare involved in these 
assessments 

2.18 

Incorporate requirement for 
contractor generated HSIP(s) 
and associated source 
selection criteria in Request for 
Proposals (RFPs) 

2.18.1 Review RFP language, CDRLsand DIDsfor 

HSI Planning activities for each Human 

Perforrmance 

2.19 

Apply analysis findings to Test 
and Evaluation Strategy (TES) 
and TEMP 

2.19.1 Special analysis to update thresholds, 
objectives, and evolving criteria for DT&E and 
eventually OT&E 

2.20' 

Conduct formal development of 
human-centered source 
selection criteria for RFPs at 
Milestone (MS) A 

2.20.1 HSI review of Source Selection Criteria for 
developmental RFPs 
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HRL 

Activity 

Sub-Activity Detaii 

Acquisition 
&Sustainment 
Tooikit Linkages 

Integration 
Notes and 
Comments 

Target 
Documents 
(e.g. ICD or 
RFP) 

Action OPR 
(generaliy 
within HSi 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL3= 

Component 
Human Touch 
Point Resolution: 
Refining 
Requirements 
Thresholds 

HRL-3 activities 
shouid occur as 
soon as possibie 
after MS A, prior to 
MS B. Nominaiiy 
about mid-way to 









3.1 

HSI Assessment - HRL3 (HSIA- 
3).With formal conversion from 
portfolio/project to Program at 
MSA, PM must assess 
completion of HRL 1 &2 
activities with emphasis on 
continuity and partnership with 
MAJCOM RM HSI WG/IPT MSA 
Phase analytic activities to date 

3.1.1 Manpower/Workload/ Functional Allocation 

3.1.2 Personnel / KSAclusters/ selection/ retention/ 
career path 

3.1.3 Training/KSAs 

3.1.4 Environment 

3.1.5 Survivability / Situation Awareness Kll Chain 
Analysis Force Protection/ Egress systems 

3.1.6 Habitabiltiy / Sustained Op^ Facilities 

3.1.7 Occupational Health 

3.1.8 Safety/ HFACs 

3.1.9 Human Factors Engineering 

3.1.9.1 Refine user needs, characteristics, and 
environmental constraints 

3.1.9.2 Support prototyping and capability based 
technology down-selection 

3.1.10 HSI WG review HSI issues log and mitigaton 
progress 

3.1.11 HSI WG specifiesand recommends risk 
matrix outcomes and HRL assignment to PM 








3.2 

Update/revise HSIP 

3.2.1 Using worksheet as described in HRL-1 to 
update sequence of activities remaining 
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3.3 

Analysis to support manpower 
estimates 

3.3.1 Emphasis on updating manning estimates as 
component HTPsare resolved 

3.3.2 Update ME data where appropriate 


3.4 

Second iteration of TSRA 

3.4.1 Update ISD analyses 

3.4.2 Explore training technology options in 
components (embedded training?; distributed 
training?) 


3.5 

Update HSI issues log 

3.5.1 Emphasis on HFE design resolutions of issues 


3.6 

Initial estimates of MPT costs 

3.6.1 Increasng detail in cost estimation process 

3.6.1.1 Participate in HSI IPT: Perform analyses to 
support development of the ME 

3.6.1.2 Refine and finalize ME 


3.7 

Syncronize with DoDAF and or 
incorporate/roll-up into SEMP 

3.7.1 Provide updated information to RM for 
continued expansion of OVs 5, 6, & 7, and CONORS; 

3.7.2 Continue specification of HV-C-1 and begin 
populating all HVs as component data evolves. 

3.7.3 Update design reference mission scenarios as 
needed. 







3.8 


Ensure HSIA updated for any 
newly introduced component 
technologies 


3.8.1 Update HSIA for any introduced component 
technology. (Especially important for commercial off 
the ^elf (COTS)) 





3.9 

Initiate Cognitive Task Analysis 
Process to inform component 
technology developments 

3.9.1 Begin component Cognitive Task Analyses 
(feeds over to 3.7.2 and 3.7.3) 

3.10. 

Update THA and HRA 

3.10.1 Update either HRA or THA or both 

3.10.2 Udate HFACsor similar nanacodes as 
components gel 

3.11 

Initiate Ergonomics/ 
biomechanical / 
anthropometric assessment 
processes 

3.11.1 Analysis of evolving HTPs taking into account 
form, fit and function to accommodate both static 
and dynamic human at work; includes accessability 
and egress during emergency situations and normal 
work 

3.12 

Plan modeling to inform 
human-in-the-loop analyses 
and assessments based on cost 
benefit 

3.12.1 Conduct analysisand negotiate specification 
of Independent Variables (IVs) for linear modeling. 
Select IVs in part based on factors that can be 
manipulated in man-in-the-loop ((MITL) usually 
simulation-based) experiments. Modeling then can 
reduce complex, multivariable solution space 
followed by iterative MITL validation in medium-to 
high-fidelity amulation 

3.13 

Update Human Machine 
Interface (HMI) assessment 

3.13.1 Analysis of interface design considerations 
primarily related to controls and displays but 
extending to any human touch point, with or without 
the use of tools and the design and diversity of the 
tools themselves 

3.14 

Conader high ROI use of mock- 
ups for physical layouts of HTPs 

3.14.1 Evaluate use of mock-ups to allow high-level 
assessment of safety, communications, workflow, 
usability, and general form and fit of design 
alternatives. These are especially useful when 
experienced legacy system operators and 
maintainers are involved in these assessments 
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3.15 

Subsystem Hazard Analyas 
(SSHA) 

3.15.1 Emerging Subsystem Hazard Analysis (SSHA) 








3.16 

Apply analysis findings to TEMP 

3.16.1 Special analysisto update thresholds, 
objectives, and evolving criteria for DT&E and OT&E 





















HRL 

Activity 

Sub-Activity Detail 

Acquisition 
&Sustainment 
Tooikit Linkages 

Integration 
Notes and 
Comments 

Target 
Documents 
(e.g. iCD or 
RFP) 

Action OPR 
(generaiiy 
within HSI 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL4= 

Component 
Human Touch 
Point 

Engineering 
Parameters and 
Human 
Performance 

HRL-4 activities 
should occur as 
soon as possible 
after MS A and prior 
to MS B. Nominally 
prior to SRR. 









4.1 

HSI Assessment ~ HRL4 (HSIA- 
4) 

4.1.1 Assessment based on best available solution 
description across all HSI domains to define level of 
risk in each. Each domain dictates critical items of 
consideration bearing on system design depending 
on the (initially likely, and later definite) HTPs in a 
system as it evolves. Risk, for purposes of HRLs, is 
mitigated by professional analytical and managerial 
activities that link critical items of consideration to 
domain-specific deagn decisions, including well- 
informed trade-offs. 

4.1.2 HSI WG review HSI issues log and mitigaton 
progress 

4.1.3 HSI WG specifies and recommends risk matrix 
outcomesand HRL assgnmentto RM/PM 
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4.2 


Update/revise HSIP 


4.2.1 Using work^eet as described in HRL-1 to 
update sequence of activities remaining 






4.3 

Update/revise LCMP 

4.3.1 Update current LCMP 


4.4 

Update Human Machine 
interface (HMi) assessment 

4.4.1 Analysis of interface design considerations 
primarily related to controls and displays but must 
be extended to any human touch point, with or 
without the use of tools and the design and diversity 
of the tools themselves. 

4.4.2 Perform technology assessments (with 
representative users) and evaluate HFE based 
technology/system requirements, user 
characteristics, and human performance 

4.4.3 Cognitive walkthroughs/HMI/HCI Evaluations/ 
Human Performance Evaluations and Risk Analyses 

4.4.4 Conduct trade studies to assess trade-offs 
between technologies 

4.4.5 Conduct trade studies to assess trade-offs 
between technologies 

4.4.6 Identify HFE technical risks and mitigation 
strategies included specification of applicable 
guides and standards 

4.4.7 Provide technology down selection 
recommendations 

4.4.8 Review sequencing and execution of 
component usability testing 
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4.5 

Consider high ROI use of mock* 
ups for physical layouts of HTPs 

4.5.1 Formally consider mock-ups to allow high- 
level assessment of safety, communications, 
workflow, usability, and general form and fit of 
deagn alternatives. These are especially useful 
when experienced legacy system operators and 
maintainers are involved in these assessments 


4.6 

Analysis of support availability 
for training needs and 
manpower / personnel 
requirements 

4.6.1 Communicate with RM and HSI functionals to 
take initial opportunity to bring critical stakeholders 
from using and supporting commands into the loop 
for planning long-term training and manpower 
support needs. Also, provides critical opportunity for 
early id and resolution of evolving training system 
gaps. 

4.6.2 Participate in technology assessments to 
evaluate adequacy of manpower estimates and 
operational impacts of proposed manning solutions. 

4.6.3 Refine/update workload for specific 
technology ops and support activities 

4.6.4 Develop manpower reduction strategy 

4.6.5 Assess manpower tradeoffs among 
technologies 

4.6.6 Participate in technology assessments to 
evaluate personnel requirements and the adequacy 
of personnel-related content in the TSP 

4.6.7 Assess personnel trade-offs between 
technologies 







4.7 


Subsystem Hazard Analyas 
(SSHA) 


4.7.1 Conduct/update Subsystem Hazard Analysis 
(SSHA) 



104 





Iterative Tranining system 
evolution 


4.8.1 Participate in technology assessments to 
evaluate adequacy of the TSP and operational 
impacts of proposed training solutions 

4.8.2 Identify and document formal training courses 
and training resource requirements 

4.8.3 Type of training curricula 

4.8.4 Ensure sufficient funding to 
support training needs,training 
device s,equipment 

4.8.5 Assess training trade-offs between 
technologies 


lApply analysisfindingsto TEMP 


4.9.1 Conduct special analysisto update thresholds, 
objectives, and evolving criteria for DT&E and OT&E 


Begin formal development of j .u. ^ 

, ” , 4.10.1 Based on requirements being translated into 

human-centered source , . . x.. ... . .. 

. , __ RFPs, begin drafting HSI-specific source selection 

selection criteria for RFPs at • » -i- 

MSB 


4.11.1 Provide updated Information to RM for 
continued expansion of OVs 5, 6, & 7, and CONORS; 
Syncronize with DoDAF and or 4.11.2 Continue specification of HV-C-1 and continue 
incorporate/roll-up into SEMP populating all HVs ascomponent data evolve& 

4.11.3 Update design reference mission scenarios as 
needed 
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HRL 

Activity 

Sub-Activity Detaii 

Acquisition 
&Sustainment 
Tooikit Linkages 

integration 
Notes and 
Comments 

Target 
Documents 
(e.g. ICD or 
RFP) 

Action OPR 
(generaily 
within HSi 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL5=Limited 
System Human 
Performance 
Parameters/ 
Demonstration 

HRL-5 activities 
shouid occur as 
soon as possibie 
after MS A and prior 
to MS B. Nominaiiy 
prior to SFR. 









5.1 

HSI Assessment - HRL5 
(HSIA-5) 

5.1.1 Assessment based on best available Solution 
Description across all HSI domains to define level of 
risk in each. Each domain dictates critical items of 
consideration bearing on sy^em design depending 
on the (initially likely, and later definite) HTPs in a 
system as it evolves. Risk, for purposes of HRLs, is 
mitigated by professional analytical and managerial 
activities that link critical items of consideration to 
domain-specific design decisions, including well- 
informed trade-offs. 

5.1.2 HSI WG review HSI Issues log and mitigaton 
progress 

5.1.3 HSI WG specifies and recommends risk mafrix 
outcomes and HRL assignment to RM/PM 








5.2 

Update/revise HSIP 

5.2.1 Using worksheet as described in HRL-1 to 
update sequence of activities remaining 
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5.3 


5.3.1 Refine/update Personnel requirements/ KPP 
and KSA in CDDs 

5.3.2 Assist in finai technology selections/Ensure 
Personnel needs can be met 
appropriately/addresses Personnel requirements 


5.4 


Confirm SOH findings 
incorporated in design and fed 
forward to sustainment 
planning 


5.4.1 Confirm that safety and Occupational Health 
design features have been incorporated 

5.4.2 Non*materiel safety and health hazard 
mitigations have been incorporated into PESHE 


5.5 


Subsystem Hazard Analysis 
(SSHA) 


5.5.1 Review component technologies and update 
Hazard Analyses 


5.6 


Update Human 
Machine/Equipment Interface 
(HMI) assessment 


5.6.1 Refine/update HFE requirements 

5.6.2 HMI/HCI usability assessments of component 
Workspaces/ Communication systems/ Human 
Performance & Safety/Occ Health featured 

5.6.3 Refine KPPs and KSAs in CDD 

5.6.4 Assist in final technology selections, assure 
solution: 

5.6.4.1 Addresses HFE requirements 

5.6.4.2 Addressesuser needsand goals 

5.6.4.3 Satisfies concept and mission objectives 

5.6.4.4 Incorporates HFE design standards 
5.6.5 Component performance testing. Update 
predictive models based on outcomes 
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5.7 


lUpdate ME 


5.7.1 Refine/update Manpower requirements 

5.7.2 Assure final technology selections 
addresses Manpower requirements 

5.7.3 Ensure MEs can be met appropriately 


5.8 


lApply analysisfindingsto TEMP 


5.8.1 Special analyss to update thresholds, 
objectives, and evolving criteria for DT&E and OT&E 


5.9 


Translate design decisions into 
deagn specifications and/or life 
cycle sustainment factors 


5.9.1 Continue formal development of human- 
centered source selection criteria for RFPsat MS B 

5.9.2 Update/revise LCMP 


5.10. 


Syncronize with DoDAF and or 
incorporate/roli-up into SEMP 


5.10.1 Syncronize with DoDAF and or 
incorporate/roli-up into SEMP 


5.11 


Update Training for design 
decisions and update 
sustainment accordingly 


5.11.1 Update Training for design decisions and 
update sustainment accordingly 


5.12 


ISynchronize with DoDAF 


5.12.1 Ensure that human reliability issues have 
been identified for action in deagn or appropriate 
process within the HV 

5.12.2 Provide updated information to RM for 
continued expansion of OVsS, 6, & 7, and CONORS; 
continue specification of HV-C-1 & D-2, and 

5.12.3 Continue populating all HVs as component 
data evolves 

5.12.4 Update design reference mission scenarios as 
needed 
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HRL 

Activity 

Sub-Activity Detail 

Acquisition 
&Sustainment 
Toolkit Linkages 

Integration 
Notes and 
Comments 

Target 
Documents 
(e.g. ICD or 
RFP) 

Action OPR 
(generaiiy 
within HSI 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL6=Field 
(Relevant 
Environment} 
Validation of 
Human 
Performance 
Prototypes 

HRL-6 activities 
shouid occur prior to 
MSB. 









6.1 

HSI Assessment - HRL6 (HSIA- 
6) 

6.1.1 Assessment based on best available Solution 
Description across all HSI domains to define level of 
risk in each. Each domain dictates critical items of 
consideration bearing on system design depending 
on the (initially likely, and later definite) HTPs in a 
system as it evolves. Risk, for purposes of HRLs, is 
mitigated by Professional analytical and managerial 
activities that link critical items of consideration to 
domain-specific design decisions, including well- 
informed trade-offs. 

6.1.2 HSI WG review HSI Issues log and mitigaton 
progress 

6.1.3 HSI WG specifies and recommends risk matrix 
outcomes and HRL assignment to RM/PM 








6.2 

Update/revise HSIP 

6.2.1 Using worksheet as described in HRL-1 to 
update sequence of activities remaining 
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6.3 


Update/revise LCMP 


6.3.1 Update/revise LCMP 






6.4 

Subsystem Hazard Analysis 
(SSHA) 

6.4.1 Subsystem Hazard Analyas (SSHA) 


6.5 

System Hazard Analysis (SHA) 

6.5.1 Perform system safety analyses 

6.5.2 Derive and finalize safety requirements and 
Criteria for Analysis, 6.5.2.1 develop preliminary 
hazard list 

6.5.2.2 Initiate preliminary hazard analysis and 
threat hazard assessment 

6.5.3 Complete initial /PESHE 


6.6 

Iterative usability testing 

6.6.1 Conduct Evalulations of Human Perfomance 
embedded in demontration system; 

6.6.2 Update predictive models based on outcomes 


6.7a 

Apply analysis findings to TEMP 

6.7.1 Special analysis to update thre^olds, 
objectives, and evolving criteria for DT&E and OT&E 

6.7.2 Participate/in the T&E/IPT 

6.7.3 Perform analyses to ensure appropriate HFE 
considerationsare included in system technology 
development// Operational /Workflow Analysis Task 
Analyses 

6.7.4 Assist in development of the system functional 
baseline and Functional Allocation Analyses 

6.7.5 Develop the HFE T&E//V&V program 
plan^/Method^Metrics /Analyses /Tools 

6.7.6 Review and provide inputs to preliminary 
designs allocated baseline 




no 





6.7b 


Apply analysis findings to TEMP 


6.7.7 Provide recommendations based on 
Manpower analyses 

6.7.8 Assess gaps between personnel capabilities 
and system needs 

6.7.9 Provide Personnel inputs to functional 
allocation 

6.7.10 Provide recommendations based on 
Personnel analyses 

6.7.11 Assess gaps between personnel capabilities 
and system needs 

6.7.12 Provide Personnel inputs to functional 
allocation 

6.7.13 Provide recommendations based on Training 
analyses 


6.8 


Complete formal development 
of human-centered source 
selection criteria for RFPsat 


6.8.1 Complete formal development of human- 
centered source selection criteria for RFPsat MS B 


MS B 


6.9.1 Provide updated information to RM for 
continued expansion of OVs 2, 3, & 5, and CONORS; 
continue specification of HV-E-1, 2, &3, HV-C-2, and 


6.9 


Syncronize with DoDAF 


HV-G. 

6.9.2 Continue populating all HVs as component 
data evolves 

6.9.3 Update design reference mission scenarios as 
needed 


Collect evidence for validation 
and verification (V&V), and 
acceptance against HSI 
requirements, including 
interoperability._ 


6.10.1 Collect evidence for validation and 
verification (V&V), and acceptance against HSI 
requirements, including interoperability 


6 . 10 . 






HRL 

Activity 

Sub-Activity Detaii 

Acquisition 
&Sustainment 
Toolkit Linkages 

Integration 
Notes and 
Comments 

Target 
Documents 
(e.g. ICD or 
RFP) 

Action OPR 
(generally 
within HSI 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL7=Final 
DT&E Human 
Performance 
Parameters 

HRL-7 activities shouid 

occur as soon as 
possible after MS B. 
Nominally by CDRA. 









7.1 

HSI Assessment ~ HRL7 (HSIA- 
7) 

7.1.1 Assessment based on best available Solution 
Description across all HSI domains to define level of 
risk in each. Each domain dictates critical items of 
consideration bearing on system design depending 
on the (initially likely, and later definite) HTPs in a 
system as it evolves Risk, for purposes of HRLs, is 
mitigated by Professional analytical and managerial 
activities that link critical items of consideration to 
domain-specific design decisions, including well- 
informed trade-offs Emphasis at this stage should be 
optimizing transfer of HSI issues and consideration 
information to the sustainment community, most 
specifically logistics center managers 

7.1.2 HSI WG review HSI Issues log and mitigaton 
progress 

7.1.3 HSI WG specifiesand recommends risk matrix 
outcomes and HRL assignment to PM 








7.2 

Update/revise HSIP 

7.2.1 Using worksheet as described in HRL-1 to 
update sequence of activitiesremaining 
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I Update/revise LCMP 


7.3.1 Assess detailed system desgns to ensure they 
meet Manpower requirements 

7.3.2 Participate in detailed system design 
demonstrations 

7.3.3 Provide recommendations for 
redesign/modification and/or develop mitigation 
strategies for Manpower estimates as required 

7.3.4 Identify potential personnel gaps based on 
detailed system deagns 

7.3.5 Update personnel requirements 

7.3.6 Participate in detailed system design 
demonstrations 


Review and update Training 
systems plans 


7.4.1 Identify potential gaps in training based on 
detailed system designs 

7.4.2 Update training requirements 

7.4.3 Update KSAs 

7.4.4 Update media analyss of training tasks 

7.4.5 Review selections to determine if still 
appropriate based on current system design/ 

7.4.6 Participate in detailed system design 
demonstrations 

7.4.7 Design training solutions (ILT / ICW/ 
Embedded training (ACAT 1)/Performance aids 

7.4.8 Develop JQR/PQS Sheet 

7.4.9 Develop and prototype training solutions 

7.4.10 Develop an assessment plan for training 
solutions 


ISystem Hazard Analysis (SHA) 


7.5.1 Assess detailed system design to ensure they 
meet SOH requirements 

7.5.2 Provide recommendations for redesign 
modification and/or develop mitigation strategies for 
SOH as required 

7.5.3 Participate in detailed system design 
demonstrations 

7.5.4 Provide recommendations for 
redesign/modification and/or develop mitigation 
strategies for Personnel qualifications as required 

7.5.5 Refine/update system safety analyses 

7.5.6 Finalize hazard analyses 
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7.6.1 Evalulations of Human Perfomance embedded 
in prototype system; 

7.6.2 Support development of detailed design 
specifications for HMI/work^ation^workspaces/ HCI/ 
Facilities 

7.6.3 Iteratively develop and review prototypes, 
mockups, screenshots, drawings and simulation^/ 
Plan and conduct cognitive walkthrough^ usability 
teeing/ heuristic evaluation^ workspace evaluations 

7.6.4 Evaluate initial system designs and 
specificationshaving an impact on human 
performance and safety 

7.6.5 Verify detailed system deagns 

7.6.6 Address HFE requirements 

7.6.7 Incorporate HFE design standards 

7.6.8 Support concept/mission goal^objective^ 
satisfy user needsand goals 

7.6.9 Mitigate known human performance risks 

7.6.10 Provide recommendations to resolve detailed 
system design deficiencies impacting users and 
system performance 

7.6.11 Update predictive models based on outcomes 


7.7.1 Special analysis to update thresholds, 
objectives, and evolving criteria for DT&E and OT&E 

7.7.2 Develop a user-centered test plan to evaluate 
initial detailed system designs 

7.7.2.1 Include operational use cases and 
operationally relevant tasks 

7.7.3 Finalize HFE requirements 

7.7.4 Determine assessment methods 
Apply analysis findings to TEMP 7.7.5 Identify metrics 

7.7.6 Develop a user-centered test plan to evaluate 
the system during component and integration testing 

7.7.7 Identify scope of test demonstra- tions and 
develop procedures, materials, and instruments 

7.7.8 Identify high risk areasfor testing 
operationally relevant use cases and tasks 

7.7.9 Operationally define human performance 
metric^ validation approaches for HFE requirements 
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7.8 

Syncronize with DoDAF 

7.8.1 Provide updated information to RM for 
continued expansion of SVs 1,2,3,6,7,8,9, 
and CONOPS; 

7.8.2 Continue specification of HV-F-2 & 3, and 
HV-G. 

7.8.3 Continue populating all other HVsas 
system evolves 








7.9 

Error and fault analysis 

7.9.1 Analysis to cover human error perormance, 
equipment operability, safety procedures and error 
recovery mechanisms. 

7.9.2 Conduct trials to support V&V, and acceptance 
against HSI requirements, including interoperability 








7.10. 

Begin formal development of 
human-centered source 
selection criteria for RFPsat 
MSC 

7.10 Begin formal development of human-centered 
source selection criteria for RFPs at MS C 








7.11 

Review and update HSI issues 
log 

7.11.1 Review and update HSI issues log 























HRL 

Activity 

Sub-Activity Detail 

Acquisition 
&Sustainment 
Tooikit Linkages 

integration 
Notes and 
Comments 

Target 
Documents 
(e.g. ICD or 
RFP) 

Action OPR 
(generaiiy 
within HSI 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL8=OT&E 

Human 

Performance 

Parameters 

HRL-8 activities 
shouid occur 
roughiy at MSC, in 
support of LRiP 
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8.1.1 Assessment based on best available Solution 
Description across all HSI domains to define level of 
risk in each. Each domain dictates critical items of 
consideration bearing on system design depending 
on the (initially likely, and later definite) HTPs in a 
system as it evolves. Risk, for purposes of HRLs, is 
mitigated by Professional analytical and managerial 
activities that link critical items of consideration to 

HSI Assessment ~ HRLS (HSIA- domain-specific design decisions, including well- 

8) informed trade-offs. Emphasis at this stage should 

be optimizing transfer of HSI issues and 
consideration information to the sustainment 
community, most specifically logistics center 
managers 

8.1.2 HSI WG review HSI issues log and mitigaton 
progress 

8.1.3 HSI WG specifies and recommends risk matrix 
outcomes and HRL assignment to PM 


Update/revise HSIP 


8.2.1 Using worksheet as described in HRL-1 to 
update sequence of activities remaining 


Update/revise HFEfor Design 
and LCMP 


8.3.1 Evaluate system componentshaving an impact 
on human performance and safety 

8.3.1.1 Identify human performance risks 

8.3.1.2 Provide HFE mitigation strategies 

8.3.2 Provide inputs to developmental test plans to 
ensure they include appropriate human 
performance assessments for integrated system 
testing to address: 

8.3.2.1 HFE requirements 

8.3.2.1.1 User interaction with HW/SW interfaces, 

8.3.2.1.2 COTS integration, and 

8.3.2.1.3 operational workflows 

8.3.2.2 HFE high risk areas 

8.3.2.3 Multiple user roles 

8.3.3 Collect human performance data during 
system test events 

8.3.3.1 HFE RequirementsCompliance Assessments 

8.3.3.2 HFE System Performance Assessments 

8.3.3.3 Usability Testing 
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I Update/revise HFEfor Design 
land LCMP 


8.3.4 Document HFE-related system performance 
issues (i.e., PTRs) 

8.3.5 Verify system compliance with HFE 
requirements 

8.3.6 Analyze results from HFE testing 

8.3.6.1 Report significance of findings based on user 
impact and system performance risk 

8.3.6.2 Incorporate impact assessment/ risk analysis 
design recommendations' Human Performance 
Mitigation Strategies 

8.3.7 Support the review and development of job 
aids 

8.3.8 Participate in ECP development 

8.3.8.1 Generate HFE related ECPs 

8.3.8.2 Review ECPs for human performance 
impacts 

8.3.9 Participate in/system test events 


Update/revise 

Manpower/Personnel inputs 
for Design and LCMP 


8.4.1 Perform analysis and reporting on Manpower 
related test events, data collected, and findings 

8.4.2 Revise the ME to reflect system design 
8.4.2.1 Provide recommendations based on findings 
to support manning as needed 

8.4.3 Perform analysisand reporting on Personnel 
related test events, data collected, and findings 


Review and update Training 
systemsfor Design and LCMP 


8.5.1 Assess training solutions during initial system 
component demonstrations 

8.5.2 Refine training solutions based on findings 
from initial system component demonstrations/ 
Trainer installations' Equipment, facility, and TTE 
plan 

8.5.3 Modify ILE 

8.5.4 Provide inputs to developmental test plans to 
ensure they Include appropriate assessments of 
training solutions for integrated system testing 

8.5.5 Participate in system test events 

8.5.6 Arrange for pilot/cadre course to train users 

8.5.6.1 Assess results from training pilot course 

8.5.6.2 Compare to learning objectives 

8.5.7 Revise training solutions based on findings 
from integrated system demonstrations and pilot 


8.5.8 Deliver final training solution for u 
production and deployment phase 


117 




8.6 

System Hazard Analysis 
(SHA)//Operating and Support 
Hazard Analysis (O&SHA) 

8.6.1 Participate in system test events 

8.6.2 Perform analyses and reporting on SOH 
related test events, data collected, and findings 

8.6.3 Revise the System Safety Analyses and Hazard 
analyses to reflect demonstrated system design 

8.6.4 Provide recommendations based on findings 
to support system design changes 


8.7 

Iterative usability testing 

8.7.1 Evalulations of Human Perfomancce 
embedded in integrated system; 

8.7.2 Update predictive models based on outcomes 


8.8 

Apply analysis findings to TEMP 

8.8.1 Special analysis to update thresholds, 
objectives, and evolving criteria for OT&E 


8.9 

Complete formal development 
of human-centered source 
selection criteria for RFPsat 
MSC 

8.9.1 Update draft HP-related source selection 
criteria, primarily from integrated prototype DT&E 
and advanced usability testing 


8.10. 

Syncronize with DoDAF 

8.10.1 Provide updated information to RM for 
continued expansion of SVs 1, 2, 3, 6, 7, 8, 9, and 
CONORS; 

8.10.2 Continue specification of HV-F-2 & 3, and HV- 
G. 

8.10.3 Continue populating all other HVsassystem 
evolves 


8.11. 

Review and update HSI issues 
log 

8.11.1 Review and update HSI issues log 






HRL 

Activity 

Sub-Activity Detail 

Acquisition 
&Sustainment 
Toolkit Linkages 

integration 
Notes and 
Comments 

Target 
Documents 
(e.g. iCD or 
RFP) 

Action OPR 
(generaily 
within HSI 

WG) 

References 

Products (of 
Activity) 

Rough Cost 
Estimate 

HRL9= 

Sustainment: 
Initiation of 
Capability Gap 
Feedback Cycle 

HRL 9 Hand-oH transition from 
Acquisition PM to 
Logistic^sustainment PM as 
keeper of HSI 

Plan/documentation. Shred 
copy to next block Acq PM. 



Emphasis at this stage 
should be optimizing 
transfer of HSI issues 
and consideration 
information to the 
sustainment 
community, most 
specifically logistics 
center managers. 






9.1 

HSI Assessment ~ HRL9 (HSIA- 

9) 

9.1.1 Assessment based on best available Solution 
Description across all HSI domains to define level of 
risk in each. Each domain dictates critical items of 
consideration bearing on system design depending 
on the (initially likely, and later definite) HTPs in a 
system as it evolves. Risk, for purposes of HRLs, is 
mitigated by Profesaonal analytical and managerial 
activities that link critical items of consideration to 
domain-specific design decisions, including well- 
informed trade-offs. 

9.1.2 HSI WG review HSI Issues log and mitigaton 
progress 

9.1.3 HSI WG specifies and recommends risk matrix 
outcomes and HRL assignment to PM 

9.1.4 Define process and responsibilities for drafting 
Program Increment HSI Transition Final Report for 
Logistic^ Sustainment PM 


Transition to 
Logistics/Sustainment 
Program Management is 
Emphasis and Keystone 
to HSIsuccess 
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IKFE 10 T&E support 


9.2 Review verify and develop KFE installation 
criteria/ 

9.2.1.1 Provide recommendations for improvement 

9.2.1.2 Consider human performance, workload, 
and safety/ Identify gaps or potential installation 
problems 

9.2.1.3 Perform analysisof LRIP deficiencies to 
Identify human performance risks 

9.2.1.4 Provide HFE mitigation strategies 

9.2.2 Develop a user-centered assessment plan to 
support OT&E considering performance quality/ 
compatibility/ interoperability/ system capabilities 

9.2.3 Participate in lOT&E 

9.2.3.1 Validate HFE-based system changes 

9.2.3.2 Collect user feedback and human 
performance data 

9.2.3.3 Report significance of findings based on user 
impact and system performance risk 

9.2.4 Review HSI ISSUE reports and discuss HSI 
migationStradeoffs 

9.2.5 Verify and validate final HFE-related 
production items 


I KFE FO T&E support 


9.3.1 Collect and analyze user feedback and service 
use data from the post-deployed system 

9.3.2 Analyze known system performance 
issues/program trouble reports 

9.3.3 Identify HFE lessons learned and life cycle 
improvements 

9.3.4 Provide ECP recommendations 

9.3.5 Provide mitigation strategies for outstanding 
HFE issues and risks 

9.3.6 Develop a user-centered assessment plan to 
support FOT&E considering performance/ quality/ 
compatibility/ interoperability/ system capabilities 

9.3.7 Participate in FOT&E 

9.3.7.1 Analyze results from FOT&E to assess and 
redesign aspects of the system/ workstation^ 
workspaces/ allocation of function^ procedure^ job 
aid^ Human-Machine Interfaced Human-Computer 
Interfaces 

9.3.8 Develop an operations and support plan to 
monitor and assess the system and collect feedback 
from users 
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HFE FOC support and 
Logistic^Sustainment 
Transition 


9.4.1 Identify/document HFE lessons learned and 
lifecycle improvements from fielded system and 
feed forward 

9.4.2 Collect feedback from operational users across 
multiple platforms and commands 

9.4.3 Perform a system assessment to identify 
capability gaps 

9.4.4 Document system areas that have been 
significantly impacted by HFE including: 
performance risk mitigation safety & health, quality 
of life, affordability 

9.4.4.1 Analyze and model post-product 
improvements for the next incremental build of 
Human-Machine Interfaces, facilities, 
workstation^workspace^ Human-Computer 
Interfaces 


HFE FOC support and 
Logistic^Sustainment 
Transition 


9.4.4.2 Provide HFE based recommendations for 
system modernization and increment upgrades 

9.4.4.3 Identify and assess design/engineering 
tradeoffs for modernization 

9.4.4.3.1 Identify advanced technologies that will 
have a significant positive impact on human 
performance 

9.4.5 Review /HSI ISSUE /reports and /discuss HSI 
/tradeoffs from an HFEdesagn impacts perspective 

9.4.6 Participate to include HFE summary in 
Program Increment HSI Transition Final Reporttor 
Logistic^ Sustainment PM 


9.5.1 Tranation Post-Fielding Training Evaluation 
Analysis (PFTEA). Validation of system training in 
operational setting and formal certification for FOC 

9.5.2 Conduct a training effectiveness 
evaluation// identify training deficiencies and root 


Training FOC support and 

Logistic^Sustainment 

Transition 


9.5.3 Provide recommendations to correct training 
deficiencies/ incorporate representative mission 
scenarios 

9.5.4 Assess human performance and effectiveness 
of training transfer 

9.9.5 Redesign training procedures and decision 


9.5.6 Identify/document training lessons learned 
and feed forward 

9.5.7 Incorporate findings into Program Increment 
HSI Transition Final Report for Logistic^Sustainment 
PM 

















The PFTEA is required 
to validate institutional 
training to ensure 
training requirements 
are met. 
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9.6 

Manpower/Personnel FOC 
support and 
Logistic^Sustainment 

Transition 

9.6.1 Verify ME survey 

9.6.2 Verify system installations meet Manpower 
requirements 

9.6.3 Assess effects of equipment installations on 
workload 

9.6.4 Identify/document manpower lessons learned 
and feed forward 

9.6.5 Identify and document areas where 

Manpower has had a significant impact 

9.6.6 Provide modernization /recom-mendations 
that would optimize manpower 

9.6.7 Collect feedback from users to 
identify/document personnel lessons learned and 
feed forward 

9.6.8 Review HSI issue reports and discuss HSI 
tradeoffs 

9.6.9 Incorporate findings into Program Increment 
HSI Transition Final Report for Logistic^Sustainment 
PM 








9.7 

SOH FOC support and 
Logistic^Sustainment 
TransitionOperating and 

Support Hazard Analysis 
(O&SHA) 

9.7.1 Verify system installations meet SOH 
requirements 

9.7.2 Finalize the SOH footprint and attributes. 
Identify/document safety and occupational health 
lessons learned and feed forward 

9.7.3 Complete a System (fielded) Safety Analysis; 
include /sustaining hazard analysis for the fielded 
system and Incorporate findings into Program 
Increment HSI Transition Final Report for 
Logistic^Sustainment PM 








9.8 

Syncronize with DoDAF 

9.8.1 Provide updated information to RM for 
continued expansion of SVs 1, 2, 3, 6, 7, 8, 9, and 
CONORS; 

9.8.2 Continue specification of HV-F-2 & 3, and HV- 
G. 

9.8.3 Continue populating all other HVs as system 
evolves 

9.8.4 Archive as part of Program Increment HSI Final 
Report for Logistic^Sustainment PM 


SVs: system interface 
description, 

communication, matrix, 
information exchange 
matrix, evolution 
description, technology 
forcast, and 
performance parameter 
matrix. Training 
requirements {HV-F-2), 
training resources (HV-F 
3), and metrics (HV-G). 
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APPENDIX G. HRL SUBJECT MATTER EXPERT EVALUATION 

QUESTIONNAIRE 


ABOUT THIS REVIEW 


You are invited to participate in a Subject Matter Expert (SME) review of the evolving 
concept of the Human Readiness Eevel (HRE). As this is the first iteration of the HRE 
framework, your valuable input and feedback will be used to refine and ultimately 
advance its ongoing definition and validation. 

Please note that all data collected in this review will be kept strictly confidential. Your 
participation in the review and your responses to the review items will not be disclosed. 
Additionally, no Personal Identifying Information (PII) will be collected in this review. 


HRL BACKGROUND & ERAMEWORK STRUCTURE 


In order to effectively translate capability needs and technology opportunities into stable, 
affordable, and well-managed acquisition programs, proper measurement and 
management of technology maturity must occur. The Technology Readiness Eevel 
(TRE) has proven to be the tool and process of choice for assessing the maturity of 
developing technologies. Yet, due mainly to its intended focus, the TRE has proven 
inappropriate to consistently capturing the human-related attributes of technology and 
their association with technology readiness. It is in response to this recognized issue that 
the Human Readiness Eevel (HRE) framework was created. 

The intended purposes of HREs are to: 

1. directly support the S&T community and Resource and Program Managers by 
providing a risk management structure and process that is similar to the TRE structure 
and process, but that explicitly links technology development to the effective design and 
specification of human-centered systems in Defense Acquisition; 

2. directly support the Requirements and Program Managers by providing a potentially 
exhaustive list of candidate activities to be tailored for each program to populate HSI 
Planning with data collection and analytical activities needed to inform requirements, 
acquisition, and sustainment documents of record; and, 

3. support the policy-driven mandate to implement HSI by identifying the most critical 
HSTspecific data collection/analysis activities as factors and elements in cost estimation 
during the Analysis of Alternatives process; this will “fund the requirement” by directly 
informing PPBS insertion to complement traditional Work Breakdown Structure-based 
program factors and elements. 

To accomplish these ends, the HRE framework must provide a time/milestone-sensitive 
roadmap of activities (explicitly linked to existing Acquisition and Sustainment Toolkits, 
for example, and related guidance and standards) that: a) detail critical organizational 
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milestones that ensure functional commitment to Human Systems Integration (HSI); and 
b) define HSI domain-sensitive data collection and analysis activities. The primary 
measure of HSI risk and maturity level is based on the execution of those milestone- 
sensitive management and analytical activities. 

At the core of the HRL concept is the activities list, the central topic for this review. In a 
later stage of HRL development, an explicit process will be defined for risk management 
and reporting. The process will define how to assign each HRL as an assessed program 
achievement value, a risk management metric. The metric will depend upon the 
determination of a scale value from 1 (HSI baselining & commitment) to 9 (capability 
gap feedback cycle). A program determined to be at HRL-1 would have achieved the 
lowest level of HSI readiness, where the initial activation of HSI commitment and 
processes occurs. By the time the program has reached HRL-9, it will have progressed 
through significant HSI analyses, requirement threshold refinements, field validations of 
human performance prototypes, and extensive developmental and operational Test and 
Evaluation (T&E) of human performance parameters. As with TREs, what constitutes an 
acceptable and appropriate Human Readiness Eevel (and so, level of programmatic risk) 
will depend upon a large number of program-specific details, most critically linked to 
where a program of record is in its life cycle. 

By structuring appropriate sequencing of HSI activities and tracking the progress of HSI 
planning and execution, the HRE distinguishes itself as a significant risk management 
tool for process owners and decision makers in Defense Acquisition. The following is 
the first evaluation of the HRL framework’s worth and usability. Based on your 
feedback, our plan is to develop an updated draft of the candidate list and framework to 
form the basis for a series of developmental workshops to “fill in the matrix” and develop 
this into as practical a tool as is possible. Your inputs will be critical to this effort. 


INSTRUCTIONS 


The first worksheet (labeled: HRL Framework Draft 1) in the attached Excel Spreadsheet 
contains the content for your review. Columns A, B & C contain HSI candidate 
activities, while columns D through J contain proposed column headings for future 
HRL framework efforts. Only for the first three columns in the intended framework 
have the cells been filled in completely. 

The following items are designed to gain feedback as to the accuracy, ease of use, and 
completeness of the HRL framework activities lists. The items are organized into five 
separate groups. The first four grouping of statements reflect the HRL’s recommended 
milestone progression in the Defense Acquisition Life Cycle. Eor instance, HRL-2 is the 
criterion threshold to be met at Milestone A, therefore the first set of items pertains to 
HRLs 1 and 2. The last set of items found in the fifth group involves the proposed 
categories for future HRL framework efforts. 
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Using the attached draft of the HRL Framework, for each item below, circle or underline 
the response that best describes your feelings. Please add comments as needed. 


HRLs 1/2 

*HRL-2 should be achieved prior to Milestone A approval. 

HRL Descriptions - Column A _ 

1. The HRL descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

2. The HRL descriptions reflect appropriate timing within the acquisition life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

3. No critical information is missing from the HRL descriptions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

Activity - Column B _ 

4. The activities and their descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

5. The activities are appropriately timed within the acquisition life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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6. No critical information is missing from the activities and their descriptions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

Sub-Activity Details - Column C _ 

7. The HRL sub-activity descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

8. The HRL sub-activity details reflect appropriate timing within the acquisition life 
cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

9. No critical items of activity are missing from the sub-activities list. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

HRLs 3/4/5/6 

*HRL - 6 should be achieved prior to Milestone B approval. 

HRL Descriptions - Column A _ 

10. The HRL descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

11. The HRL descriptions reflect appropriate timing within the acquisition life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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12. No critical information is missing from the HRL descriptions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 
Comments 


Activity: Column B _ 

13. The activities and their descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 
Comments 

14. The activities are appropriately timed within the acquisition life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

15. No critical information is missing from the activities and their descriptions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

Sub-Activity Details: Column C _ 

16. The HRL sub-activity descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

17. The HRL sub-activity details reflect appropriate timing within the acquisition life 
cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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18. No critical items of activity are missing from the sub-activities list. 
Strongly Agree Agree Neutral Disagree Strongly Disagree 
Comments 


HRLs 7/8 

*HRL - 8 should be achieved prior to Milestone C approval. 

HRL Descriptions: Column A _ 

19. The HRL descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

20. The HRL descriptions reflect appropriate timing within the acquisition life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

21. No critical information is missing from the HRL descriptions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

Activity; Column B _ 

22. The activities and their descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

23. The activities are appropriately timed within the acquisition life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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24. No critical information is missing from the activities and their descriptions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

Sub-Activity Details: Column C _ 

25. The HRL sub-activity descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

26. The HRL sub-activity details reflect appropriate timing within the acquisition life 
cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

27. No critical items of activity are missing from the sub-activities list. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

HRL 9 

*HRL - 9 represents the highest HRL to be achieved and signifies the initiation of the 
capability gap feedback cycle. 

HRL Descriptions: Column A _ 

28. The HRL descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

29. The HRL descriptions reflect appropriate timing within the acquisition life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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30. No critical information is missing from the HRL descriptions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

Activity: Column B _ 

31. The activities and their descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

32. The activities are appropriately timed within the acquisition life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

33. No critical information is missing from the activities and their descriptions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

Sub-Activity Details: Column C _ 

34. The HRL sub-activity descriptions are clearly understood. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

35. The HRL sub-activity details reflect appropriate timing within the acquisition life 
cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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36. No critical items of activity are missing from the sub-activities list. 
Strongly Agree Agree Neutral Disagree Strongly Disagree 
Comments 


Proposed Categories (Columns) for Future HRL Framework Efforts 

A & S Toolkit; Column D _ 

37. This column heading and information in the column cells will he very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

38. This column heading and information in the column cells will he very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 


Integration Notes and Comments: Column E _ 

39. This column heading and information in the column cells will he very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

40. This column heading and information in the column cells will he very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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Target Documents: Column F _ 

41. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 


42. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 


Action OPR; Column G _ 

43. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 


44. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 


References; Column H _ 

45. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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46. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 


Products of Activity; Column I _ 

47. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 


48. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

Rough Cost Estimate (person days and $): Column J _ 

49. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

50. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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51. You would recommend no additional column topics to populate the matrix. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments; please list any additions you would recommend and your rationale, if 
appropriate 

#51 is the last rating item summarizing your review. Please add any comments, issues, or 
concerns below. Thank you for your time and effort in support of this work in progress. 
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APPENDIX H. HRL SUBJECT MATTER EXPERT CASE STUDY 

QUESTIONNAIRE 


ABOUT THIS REVIEW 


You are invited as key HSI practitioners in a program office to participate in a review of 
activity lists and matrix headings contained in the first installment of the evolving 
concept of the Human Readiness Level (HRL). As this is the first iteration of the HRL 
framework, your valuable input and feedback will be used to refine and ultimately 
advance its ongoing definition and validation. 

Please note that all data collected in this review will be kept strictly confidential. Your 
participation in the review and your responses to the review items will not be disclosed. 
Additionally, no Personal Identifying Information (PII) will be collected in this review. 


HRL BACKGROUND & ERAMEWORK STRUCTURE 


In order to effectively translate capability needs and technology opportunities into stable, 
affordable, and well-managed acquisition programs, proper measurement and 
management of technology maturity must occur. The Technology Readiness Level 
(TRL) has proven to be the tool and process of choice for assessing the maturity of 
developing technologies. Yet, due mainly to its intended focus, the TRL has proven 
inappropriate to consistently capturing the human-related attributes of technology and 
their association with technology readiness. It is in response to this recognized issue that 
the Human Readiness Level (HRL) framework was created. 

The intended purposes of HRLs are to: 

1. directly support the S&T community and Resource and Program Managers by 
providing a risk management structure and process that is similar to the TRL structure 
and process, but that explicitly links technology development to the effective design and 
specification of human-centered systems in Defense Acquisition; 

2. directly support Requirements and Program Managers by providing a potentially 
exhaustive list of candidate HSI-specific data collection and analytical activities to be 
tailored for each program to populate HSI Planning to inform requirements, acquisition, 
and sustainment documents of record; and, 

3. support the policy-driven mandate to implement HSI by identifying the most critical 
HSI-specific data collection/analysis activities as factors and elements in cost estimation 
during the Analysis of Alternatives process; this will “fund the requirement” by directly 
informing Planning, Programming, and Budgeting System (PPBS) insertion to 
complement traditional Work Breakdown Structure-based program factors and elements. 
To accomplish these ends, the HRL framework must provide a time/milestone-sensitive 
roadmap of activities (explicitly linked to existing System Engineering processes. 
Acquisition and Sustainment Toolkits, for example, and related guidance and standards) 
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that: a) detail critical organizational milestones that ensure functional commitment to 
Human Systems Integration (HSI); and b) define HSI domain-sensitive data collection 
and analysis activities and a clear process for their management. The primary measure of 
HSI risk and maturity level will be based on the execution of those milestone-sensitive 
management and analytical activities. 

At the core of the HRL concept is the activities list, the central topic for this review. In a 
later stage of HRL development, an explicit process will be defined for risk management 
and reporting. The process will define how to assign each HRL as an assessed program 
achievement value, a risk management metric. The metric will depend upon the 
determination of a scale value from 1 (HSI baselining & commitment) to 9 (capability 
gap feedback cycle). A program determined to be at HRL-1 would have achieved the 
lowest level of HSI readiness, where the initial activation of HSI commitment and 
processes occurs. By the time the program has reached HRL-9, it will have progressed 
through significant HSI analyses, requirement threshold refinements, field validations of 
human performance prototypes, and extensive developmental and operational Test and 
Evaluation (T&E) of human performance parameters. As with TREs, what constitutes an 
acceptable and appropriate Human Readiness Eevel (and so, level of programmatic risk) 
will depend upon a large number of program-specific details, most critically linked to 
where a program of record is in its life cycle. 

The key to a successful HSI acquisition strategy is having an accurate HSI Plan 
(HSIP) in place. The following is the first evaluation of the HRL framework’s ability 
to directly support the creation and execution of an HSIP. Based on your feedback, 
our plan is to develop an updated draft of the activity candidate list and framework to 
form the basis for a series of developmental workshops to “fill in the matrix” and develop 
this into as practical a tool as is possible. Your inputs will be critical to this effort. 


INSTRUCTIONS 


The first worksheet (labeled: HRL Framework Draft 1) in the attached Excel Spreadsheet 
contains the content for your review. Columns A, B & C contain HSI candidate 
activities, while columns D through J contain proposed column headings for future 
HRL framework efforts. Only for the first three columns in the intended framework 
have the cells been filled in completely. 

The following items are designed to gain feedback as to the ease of use, and completeness 
of the HRL framework when used to support the creation and execution of an HSIP. Eor 
each item below, circle or underline the response that best describes your feelings. Please 
add comments as needed. 
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1. The milestone-sensitive HSI activities listed and the HRL matrix (once completed) 
will provide an effective framework for an HSI working group to tailor for 
comprehensive HSI planning. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

2. The HRL framework contains all of the appropriate HSI activities that would be 
necessary to manage, create, and sustain an HSIP throughout a program’s life cycle. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

3. Using the HRL framework as an HSI maturity metric can benefit HSI risk 
identification and management within the HSIP. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 

4. The HRL framework facilitates easier HSI issue-tracking and HSI execution 
accountability. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments 
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5. The column headings in the spreadsheet (after columns A, B & C being reviewed 
above) represent a complete list of the key areas of information needed to perform 
effective HSI Planning. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 
Comments (please recommend any additional headings you think are needed) 


6. The column headings in the spreadsheet (after columns A, B & C being reviewed 
above) represent the most relevant list of the key areas of information needed to perform 
effective HSI Planning. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Comments (please identify headings you think might be eliminated as redundant or 
irrelevant) 


7. The following section is designed to gain feedback as to the relative value/importance 
of the current (columns A, B & C) and proposed categories (columns D - J) contained in 
the HRL framework when being used to support an HSIP. Please assign a rank to each of 
the ten categories listed below in order of value/importance to HSI Planning. (“1” is most 
valuable/important; tied values in the ranking are acceptable; please comment where 
appropriate; and, especially, specify any additional columns you believe need to be added 
or any current heading(s) that you feel should be eliminated) _ 


a. HRL - 


b. Activity - 


c. Sub Activity Detail - 


d. Acquisition & Sustainment (A & S) Toolkit Linkages - 


e. Integration Notes & Comments - 
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f. Target Documents - 


g. Action OPR - 


h. References - 


i. Products (of Activity) - 


j. Rough Cost Estimate - 


7. j. is the last rating item summarizing your review. Please add any additional 
comments, issues, or concerns below. Thank you for your time and effort in support of 
this work in progress. 
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APPENDIX 1. HRL SUBJECT MATTER EXPERT EVALUATION 

COMMENTS 


HRLs 1/2 

*HRL-2 should be achieved prior to Milestone A approval. 


HRL Descriptions - Column A _ 

1. The HRL descriptions are clearly understood. 

“Could be better, but not bad” 

“The description for HRL 1 is rather clear and straight forward. However for HRL 2, 
reference to “component technology development” is not real clear. Since HRL 2 is in 
support of MS A and subsequent Technology Development Phase, suggest something like 
“HSI analysis suite in support of AoA” or “HSI analysis suite in support of MS A” or 
“HSI analysis suite in support of proceeding to Technology Development Phase. ” 

“1 is relatively clear, but the only issue is this can be done at any phase based on when 
HSI involvement is initiated. I understand that the intent is to begin involvement pre MS 
A, but even if you don’t this activity can be accomplished at any point HSI gets involved. 

2 is somewhat less clear to me. I’m not sure what the suite is referring to, is it a toolkit or 
checklist? It may become more clear as I read through the information. ” 


2. The HRL descriptions reflect appropriate timing within the acquisition life cycle. 

“Getting a commitment and actually getting engaged will be very hard prior to having an 
established program. Granted - ideally this should be engaged with DP and CCTD, etc. 
How much you can actually accomplish before MS B will be challenging. Being 
informed by CBA and GBP and other analyses may provide help. ” 

“Timing within acquisition cycle cannot be ascertained from Column A descriptions. ” 

“Strongly agreed, it is very important to begin as early as possible in the process. Only 
question is what happens when you ’re starting after MS A ? Do you still go through these 
or should they have already been done? It may become OBE in the future, but I’m 
thinking about programs that are currently in process. ” 
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3. No critical information is missing from the HRL descriptions. 

“Appears to be comprehensive. Need more time to review. ” 

“The description for HRL 2 does not relay ultimate performance objective. As read, the 
objective for HRL 2 is to keep HSI practitioners employed. Besides shortfalls mentioned 
above, descriptors should describe the ultimate outcome or goal for the HRL. ” 

“I would like to understand what the suite is in HRL 2. I know you ’re planning on doing 
this in the future, but identify where each HRL is tied to TRLs and MRLs to better support 
the technology being developed, since HRLs are more process focused and TRLs/MRLs 
are product focused. ” 


Activity - Column B _ 

4. The activities and their descriptions are clearly understood. 

“References to “design” and “system” are out of place this early in the acquisition 
process where you’re considering concept alternatives (materiel and non-materiel). 
Some of the activities have same title (ex ll.a and ll.b). Recommend activity state what 
it is that is to be performed. Begin each activity with “Perform”, “Conduct”, 
“Generate”, “List”, etc.” 

“1 - Most of them are easily understood but I’m not sure how well they will be 
understood by non-HSI individuals. 1.2 and thinkyou may want to differentiate Lila and 
b and L12a and b more the detail shouldn’t be different if it’s the same activity at the 
same stage. ” 

“Activity listing format will make it hard for future development of cost elements. Can 
have a descriptor column for detail. ” 


5. The activities are appropriately timed within the acquisition life cycle. 

“Timing for phase adequate. ” 

“References to “design” and “system” are out of place this early in the acquisition 
process where you’re considering concept alternatives (materiel and non-materiel). ” 

“Some of the documents are not required at MS A. But 1 agree that the other efforts 
should begin at this stage to ensure effective HSI. ” 

“These timings are probably program-dependent, but they seem reasonably aligned with 
the DODl 5000.02 timeline, but 1 am unclear as to what processes are supporting CJCSl 
3170 activities that proceed the acquisition program. Current information indicates that 
HSI SMEs need to be operating in support of potential programs to bring forward 
lessons learned and early risk mitigation issues well before Milestone A. ” _ 
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6. No critical information is missing from the activities and their descriptions. 

“Needs further scrutiny. ” 

“We don't know what we don't know. We may find that there are critical drivers that are 
not under our purview or within our awareness on any given program, or FoS/SoS. This 
framework will probably best be constructed as an iterative model that accommodates 
periodic modifications (additions / deletions). ” 


Sub-Activity Details - Column C _ 

7. The HRL sub-activity descriptions are clearly understood. 

“Under 2.4.5 - don’t understand inclusion of reference to 2.4.1 training situation 
analysis (TSA). Either numbering is wrong or information has been inserted in the 
wrong place. ” 

“I like how they ’re broken into steps. Makes them easier to follow and accomplish each 
activity. ’’ 


8. The HRL sub-activity details reflect appropriate timing within the acquisition life 
cycle. 

“As mentioned above with the Activity detail. Some of the documents/efforts may not be 
in the appropriate place but I think it will take trying it out on a program to really see 
what can be accomplished pre MS A. At this level I can’t think of specifics besides some 
of the documents, but there may be some modeling that can’t be done as fully as 
anticipated per the descriptions. ’’ 

“Probably realistically a little early on many of them. May start before A, but unlikely to 
have the necessary funding, fidelity and support necessary to take all the tasks to 
completion or any real depth. ’’ 

“1.6.4 Difficult to estimate cost until manpower is more thoroughly explored as in 1.8.6. 
What is the difference between 1.8.4 and 1.11.2?’’ 

“I think as we examine the data that will populate columns d, f, & g timing will become 
clearer. There are activities that may be premature other than a outline for future phases 
and other activities that must be done very early in the JCIDS process or completed as 
continuous underlying HSI S&T activities that are pulled off the shelf into the JCIDS 
process in order to effectively impact (timely input, already accepted HSI requirements) a 
specific material solution. The cycle time is too short to undertake some HSI 
analyses/activities and drive requirements & functional allocation decisions. ’’ 
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9. No critical items of activity are missing from the sub-aetivities list. 

“Needs further scrutiny — appears to be comprehensive. ” 

“Unable to determine. The answer will become more apparent over time in application of 
the HRL process. ” 


HRLs 3/4/5/6 

*HRL - 6 should be aehieved prior to Milestone B approval. 


HRL Descriptions - Column A _ 

10. The HRL deseriptions are elearly understood. 

“Reference to “component” in HRLs 3&4 can mislead that system level was bypassed or 
circumvented. Not clear what is “touch point resolution ” in HRLs 3 &4.” 

“I’m not sure that the prototypes are going to be completely operational by MS B. I 
know the intent is to follow along with the TRLs but HRL 6 with the current definition 
may be somewhat optimistic. ” 


11. The HRL deseriptions refleet appropriate timing within the aequisition life cyele. 
“Timing within acquisition cycle cannot be ascertained from Column A descriptions. ” 


12. No eritieal information is missing from the HRL deseriptions. 

“Needs more scrutiny. ” 

“Descriptors should better distinguish maturity levels, particularly HRLs 3 & 4.” 

“This should be discussed in more detail at the workshop. But for now I think top level is 
well defined. ” 


Activity: Column B _ 

13. The aetivities and their deseriptions are elearly understood. 

“Could provide some more explanation / description. ” 

“It is difficult to assess sufficiency of activity descriptions when it is unclear when the 
milestone points are unclear. ” 
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14. The activities are appropriately timed within the acquisition life cycle. 

“. ..don’t assume program start at MS A -- normally not formalized till much later. ” 

“Stating HRLs 3-5 all should occur “as soon as possible after MS A” does not help 
discriminate between the three. Timing for activity completion should be more defined. 
Recommend tailoring HRL-4 activities/attainment by SRR/RFP completion. Recommend 
tailoring HRL-S activities/attainment by source selection/SFR completion. Recommend 
tailoring HRL-6 activities/attainment by PDR completion (before or after MS B). ” 


15. No critical information is missing from the activities and their descriptions. 
“Needs more scrutiny. ” 


Sub-Activity Details; Column C _ 

16. The HRL sub-activity descriptions are clearly understood. 

“HTPs not well understood / articulated throughout. ” 

“Further detail needs to be provided for some domains. I.E. Safety / Occupational 
Health Activities. Yes they looked at but actual activity is not the domain. ” 


17. The HRL sub-activity details reflect appropriate timing within the acquisition life 
cycle. 


18. No critical items of activity are missing from the sub-activities list. 
“Add 3.6. L 3 Inform cost estimate from manpower estimate. ” 


HRLs 7/8 

*HRL - 8 should be achieved prior to Milestone C approval. 


HRL Descriptions: Column A _ 

19. The HRL descriptions are clearly understood. 


20. The HRL descriptions reflect appropriate timing within the acquisition life cycle. 


21. No critical information is missing from the HRL descriptions. 
“Need to incorporate HSI trade-offs and integration of the domains. ” 
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Activity: Column B _ 

22. The activities and their descriptions are clearly understood. 


23. The activities are appropriately timed within the acquisition life cycle. 

“The timing for HRL-7 activities seems appropriate. Difficult to ascertain timing for 
“update/revise HFEfor design and LCMP” references. Reference to source selection by 
TRIP is out of place. Generally, PM may exercise option on the existing contract for 
contractor to produce TRIP units. ” 


24. No critical information is missing from the activities and their descriptions. 

“The analysis by CDR for HRL-7 needs to be pretty comprehensive as the design is pretty 
much locked in after this point and changes become difficult to influence/implement. 
Appears activities for HRL-8 should be better tailored toward what resulted from DT&E, 
and how will deficiencies be addressed prior to OT&E. ” 

“You may want to add additional information about DT&E and OT&E participation and 
the DRs associated with these tests. Also, include inputs/final analysis and trade-offs 
important to design reviews. Some of these may be based on financial implications but 
that needs to be identified. Final opportunity to influence design. ” 


Sub-Activity Details: Column C _ 

25. The HRL sub-activity descriptions are clearly understood. 

“Generally understand descriptions, but some should be improved. Activities related to 
“participate ” are rather unclear. ” 


26. The HRL sub-activity details reflect appropriate timing within the acquisition life 
cycle. 

“The timing for activities generally seem appropriate. References to providing input to 
development test plans (8.3.2) and source selection criteria (8.9.1) are too late in 
process. ” 


27. No critical items of activity are missing from the sub-activities list. 

“Need to find a way to take findings from DT&E and inform / support OT&E test 
readiness evaluation / report. Almost all HRL testing can be accomplished unobtrusively 
in DT&E and in enough detail to know what to expect in OT&E. HRL determination in 
OT&E will have to be a natural outcome from the actual tests - it must be unobtrusive. 
Also may want to get involved in OAs and EOAs to refine HRL 6 info in preparation for 
final DT&E parameters. ” 
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HRL9 

*HRL - 9 represents the highest HRL to be achieved and signifies the initiation of the 
capability gap feedback cycle. 


HRL Descriptions: Column A _ 

28. The HRL descriptions are clearly understood. 

“Recommend this HRL/description be aligned with Post Implementation Review. ” 


29. The HRL descriptions reflect appropriate timing within the acquisition life cycle. 


30. No critical information is missing from the HRL descriptions. 


Activity: Column B _ 

31. The activities and their descriptions are clearly understood. 


32. The activities are appropriately timed within the acquisition life cycle. 


33. No critical information is missing from the activities and their descriptions. 


Sub-Activity Details: Column C _ 

34. The HRL sub-activity descriptions are clearly understood. 


35. The HRL sub-activity details reflect appropriate timing within the acquisition life 
cycle. 


36. No critical items of activity are missing from the sub-activities list. 

“It is important to include feedback throughout the entire acq process, not just during 
HRL 9. ” 
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Proposed Categories (Columns) for Future HRL Framework Efforts 


Acquisition & Sustainment Toolkit Linkages; Column D _ 

37. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

“It will likely be critical for more than HRL 9 only. Since Logistics considerations 
should, like manpower and training issues, be dealt with early, it is not unlikely that 
much lower HRL levels could be influenced by the ASTK. ” 


38. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 


Integration Notes and Comments; Column E _ 

39. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

“Agreed, additional details will be especially useful for individuals tasked with HSI that 
are not very familiar with it or with the acquisition process. ” 


40. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 


Target Documents: Column F _ 

41. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

“Agreed, individuals can go look at those documents for additional detail on program. ” 

“Recommend considering list of target documents for NEXT phase here as well to 
highlight upcoming tasks for planning purposes. ” 


42. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 
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Action OPR; Column G _ 

43. This column heading and information in the column cells will he very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

“There should be a program office OPR, a Sponsor/MAJCOM OPR, and an HSI OPR. ” 


44. This column heading and information in the column cells will he very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

“Especially when individuals move on, it is nice to have a POC to go back to or identify 
that the work has been in process. ” 


References; Column H _ 

45. This column heading and information in the column cells will he very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 


46. This column heading and information in the column cells will he very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

“Will provide information to individuals performing the tasks on exactly what they need 
to do and where to go to find information. ” 

“Consider hyper linking or a separate reference database/dictionary. ” 


Products of Activity: Column I _ 

47. This column heading and information in the column cells will he very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

“Strongly agree... for building off of effort in next HRL or activity ” 


48. This column heading and information in the column cells will he very useful in 
the HRL framework matrix for all 9 HRL activity listings. 
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Rough Cost Estimate; Column J _ 

49. This column heading and information in the column cells will be very relevant 
in the HRL framework matrix for all 9 HRL activity listings. 

“Provides PM with information on the cost of doing the HSI at each level/activity for a 
program. ” 

“Agree with the spirit of what this is attempting to accomplish. I suspect that a cost 
estimate is not necessary for every single sub-activity. There are a lot of government 
activities that are taken out of hide, like commenting on a CDD or HP writing a HSIP. 
Eventually, you probably should highlight the ones you especially need the input for cost 
estimations. Since this spreadsheet lists activities to be completed by government HSI 
WG, then the cost estimate apparently will capture costs associated government 
management cost portion of the pie. ” 

“Since all the activities have such variability depending on the program, advocacy of 
PM, etc. Pm wondering what validity the estimate will have. ” 


50. This column heading and information in the column cells will be very useful in 
the HRL framework matrix for all 9 HRL activity listings. 

“Will allow PM to pick and choose the most important activities based on budget 
restrictions. ” 

“If cost is an ultimate part of goal. Possible listing in effort or man-hours versus person 
days would be relevant. Depending of labor mix, cost does not usually occur per day 
(i.e., contract personnel). Also might be good Ideas to include cost drivers or cost 
estimation relationship columns for future sourcing and appropriations. ” 


51. You would recommend no additional column topics to populate the matrix. 

“Relevant to Cost, more information such as system, and drivers may be important. ROI 
Benefit Analysis and Trade-off columns may also be useful. Common language as used 
in acquisition or reports should be used for search relationships and connections ( I.e. 
target document is good). ” _ 
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